| 1 | L4-agnostic core, L7 only as optional read-only sniffers | one proxy for all games; no protocol reverse-engineering in the core | per-game proxy; L7 termination |
| 2 | Thread-per-core + SO_REUSEPORT, thread-local session state | no locks on the hot path; linear scaling | a global session map with sharding locks |
| 3 | Immutable config snapshot + atomic swap | reload without blocking the data path | an RWLock on the live config |
| 4 | No shared session state between instances (v1) | complexity/latency; anycast reconnect is enough | Raft/Redis session store |
| 4a | (v2, ch. 10) Config and operator intent become fleet-shared via an ordered durable store (Tier 1); backend health becomes shared within a failure domain via gossip (Tier 2). Sessions stay unshared — ADR 4 holds for sessions. | operators need one authoritative config, persisted intent, and multi-vantage health; sessions do not benefit enough to justify the coupling | one global store for everything (cross-region health is noise); keep everything per-instance (intent lost on restart, not fleet-consistent) |
| 5 | Client-IP preservation optional (PROXY protocol or TPROXY) | different backend capabilities / network setups | forcing a single method |
| 6 | External routing logic via a resolver callback + cache | matchmaking stays outside; the proxy understands no tokens | hard-coding routing rules in the proxy |
| 7 | Rust | GC-free latency, safe parsers | Go (GC), C++ (memory safety) |
| 8 | UDP: a connect(2) socket per session | return path without a table lookup, kernel sender filtering | one socket + manual demux |
| 9 | UDP v0: one spawned reply-pump task per session; idle expiry by a 1 s sweep (the first cut also kept a bounded sticky table; removed in #56, affinity is a consistent_hash pool) | ships the vertical slice without recvmmsg/timing-wheel/LRU machinery; each is a drop-in later | building the batching + timing wheel + LRU up front |
| 10 | dst prefix listener: one wildcard IP_PKTINFO / IPV6_RECVPKTINFO socket per prefix, real dest per datagram, reply source via sendmsg cmsg — using nix (safe wrappers) | keeps the zero-unsafe invariant; nix was already slated for recvmmsg / TPROXY / splice | one socket bound per address (does not scale); raw libc + unsafe cmsg walking |
| 11 | Graceful shutdown drain: a watch<usize> connection counter (ConnTracker / ConnGuard) + wait_for(==0), bounded by shutdown_grace_sec; UDP recv loop enters a drain state instead of returning | no new dep; race-free wait; per-conn cost is one send_modify on open/close, nothing per byte | tokio_util::task::TaskTracker + CancellationToken (extra dep); a global Mutex<HashSet<JoinHandle>> |
| 12 | Transparent mode (TPROXY): per-listener transparent: bool. TCP — IP_TRANSPARENT on the listen socket + a client-ip:port-bound IP_TRANSPARENT upstream TcpSocket. UDP — IP_TRANSPARENT + IP_RECVORIGDSTADDR on the listen socket (recvmsg for the original ip:port), a client-bound IP_TRANSPARENT upstream socket, and a per-session IP_TRANSPARENT reply socket bound to the original destination. IP_TRANSPARENT via socket2 SockRef (bumped 0.5 → 0.6 for set_ip_transparent_v6), origdst cmsg via nix — still zero unsafe | one flag turns on the whole path; the reply socket restores source ip:port exactly (a pktinfo cmsg can only set the source IP, not the port); socket2 0.6 was already in the tree via hyper-util | PROXY protocol only (needs backend support); an IPV6_TRANSPARENT raw libc + unsafe setsockopt; reusing the prefix-mode sendmsg cmsg reply path (wrong source port) |
| 12a | Backend discovery: a level-triggered BackendSource seam in wayhouse-core (fetch → Vec<SocketAddr> = the current set; the runtime diffs it, like listener reconcile), one control-plane refresh_loop task per source, results fed through Snapshot::build_with_sources on the existing reload path (one snapshot writer). Concrete clients (hickory-resolver for SRV, reqwest for Consul / k8s) in the wayhouse binary; k8s lists EndpointSlices on the poll interval and additionally watches them, and Consul holds a blocking query on its list index (BackendSource::changed wakes refresh_loop early; the tick is the resync net; refresh_loop coalesces a burst of signals into one fetch). An errored / empty refresh keeps the last-known-good set. | matches the immutable-Snapshot invariant and lets a Tier-1 store (ch. 10) reuse the same reconcile; degrade-in-place instead of draining a pool on a discovery blip; HTTP / DNS deps stay out of wayhouse-core | add/remove deltas from each source (drift-prone); a full k8s informer that owns the cache and feeds events straight into the set (the watch only signals; fetch stays the single source of truth, so a lost event costs latency, never correctness); clearing the pool on an empty result |
| 13 | (v2, ch. 10) Level-triggered, pull-based config distribution: instances subscribe to the Tier-1 store, receive a snapshot + revision cursor + change stream, and feed each revision through the existing validate() → Snapshot::build → ArcSwap::store path. The store is one more writer; wayhouse-core is untouched, the client lives in the wayhouse binary (same seam as resolvers). | an instance offline for a while catches up from its cursor with no writer-side delivery state; a bad revision is rejected exactly like a bad file reload | push distribution (needs per-instance delivery tracking); a new data-plane config transport replacing the file |
| 14 | (v2, ch. 10) A separate wayhouse-controller: Tier-1 writer + fleet read-aggregator + the GUI’s only backend + all authn/authz/audit. Proxies never peer through it. HA = N replicas behind a leader lock for writes; its outage freezes changes, not traffic. | keeps consensus / auth off the data-plane nodes; one place to secure and audit | leader election among the proxy instances themselves; auth on every proxy admin API |
| 15 | (v2, ch. 10) Regional health authority: local active/passive checks decide; the failure-domain gossip view is quorum-weighted advice — unhealthy on local fall or domain quorum-down, healthy only on local rise, Tier-1 force-down overrides. | one flapping instance can’t poison the pool (quorum gate on “down”); one instance can’t ignore a domain-wide outage (the or); a stale remote “up” can’t revive a locally-unreachable backend | trust the shared verdict outright; share health globally (noise across reachability classes) |
| 16 | (phase 9) Sniffers run as wasmtime core modules with no WASI and a 3-function ABI (memory, alloc, sniff(…)->packed u64 returning an encoded RouteHint; the sniff signature is widened by ADR 16a). Bounds: epoch-interruption deadline (call_timeout_ms, host ticker thread) + StoreLimits memory cap; input already ≤ the route’s peek_len(). wasmtime is a wayhouse-binary dep; the Sniffer trait + a runtime-threaded Arc<Sniffers> registry stay in wayhouse-core. First-party a2s / minecraft / regex-firstbytes sniffers ship built the same way. | game-specific parsing must never be in the agnostic core or a fork; no WASI ⇒ a sniffer can’t reach fs / clock / net; epoch interruption pre-empts a runaway parser; the registry seam mirrors resolvers so wayhouse-core gains no sniffer runtime | compile sniffers in (fork per game); extism (heavier, less control over fuel/epoch); the component model + WASI (larger surface, bigger dep); an out-of-process sniffer per connection (latency — kept only as the fallback if the in-process boundary misses NFR N1) |
| 16a | (phase 9, amends ADR 16) Per-sniffer config via a widened sniff signature. sniff(in_ptr, in_len) → sniff(in_ptr, in_len, cfg_ptr, cfg_len); the host allocs + writes the module’s settings.sniffers.modules[].config string into a second linear-memory region on every call (a fresh Store per call, so it is re-marshalled each time), cfg_len == 0 when unset. config is baked onto the WasmSniffer at load/rescan time; the wayhouse-core Sniffer trait is unchanged (config is host-side). | the ABI is a third-party contract that freezes at 1.0 and there are zero external sniffers today, so the clean break is at its cheapest now; a uniform 4-arg signature (config always passed) has no “call configure if the module exports it” branch and no two-class sniffer model; passing config as a second (ptr,len) region reuses the exact mechanism the input already uses | an optional configure(ptr,len) export (compat-shim shape the pre-1.0 guardrail rejects; permanent host branch; two sniffer classes); a length-prefixed cfg blob prepended to the input region (overloads one region with two meanings; every sniffer must parse the framing even to ignore it); config-without-pin as a first-class path (deferred — a modules[] entry still requires its sha256) |
| 17 | (perf pass) TCP pump uses splice(2) on Linux, behind the same proxy::pump fn. Each direction owns one O_NONBLOCK pipe pair for the connection’s life; bytes move socket → pipe → socket via nix::fcntl::splice (feature zerocopy) driven by TcpStream::try_io readiness. Non-Linux — and a pipe2 failure — fall back to the buffered try_read / try_write loop (also over &TcpStream, no into_split). Half-close is propagated with socket2 SockRef::shutdown(Write). | no userspace copy and no 32 KiB×2 per-connection heap buffers on the steady path — the win is CPU / memory-bandwidth / RSS at high connection counts (gates N3–N5), not loopback RTT; drop-in behind the existing fn so every forwarding test still covers it | keep the buffered copy (a userspace bounce per byte); sendfile (file→socket only); io_uring (bigger dep + runtime — revisit if the pipe hop shows up in a profile); one shared pipe for both directions (couples the two directions’ backpressure) |
| 18 | (perf pass) UDP ingress batches with recvmmsg(2) (nix, Linux) — one syscall per readiness wakeup pulls up to RECV_BATCH (16) datagrams into a per-worker RecvBatch (16 × 64 KiB buffers, ~1 MiB/worker); the existing per-datagram route / filter / forward loop then runs over the batch. A fresh MultiHeaders per call (nix does not reset msg_namelen / msg_controllen between calls, so reuse would truncate a v4→v6 address or a cmsg). Non-Linux keeps one recvmsg per call behind the same RecvBatch API. The prefix / transparent dest cmsg parse moved into the batch path (dst_from_cmsg). | the listen socket is the single contended c2s funnel — amortising its recv syscall is the highest-leverage UDP change; buffers are per-worker not per-session, so no RSS blow-up; drop-in behind RecvBatch so every UDP forwarding test still covers it | sendmmsg egress batching in the same slice (per-session reply buffers would 16× the RSS, or force per-datagram allocs — deferred as its own change); reusing one MultiHeaders (address / cmsg truncation footgun); io_uring multishot recv (bigger dep) |
| 19 | (perf pass) UDP idle expiry is a single-level timing wheel (IdleWheel, WHEEL_SLOTS=512 one-second slots). A session is filed in the slot for its idle-deadline second; a tick drains one slot, evicting entries whose live last_ms shows real idleness and re-filing the rest (a datagram moved the deadline, or idle_ms > the wheel span). Exactly one entry per live session. The per-datagram hot path is unchanged — it still only bumps the atomic last_ms. | tick cost drops from O(sessions) (the old 1 s retain scan) to O(slot + re-files); activity refresh stays lazy so the datagram path pays nothing; a too-long idle_ms degrades to a re-check every WHEEL_SLOTS s, never a miss | per-datagram wheel re-insertion (moves work onto the hot path); a hierarchical wheel (more code, unneeded at 1 s granularity / ~9 min span); a BinaryHeap by deadline (O(log n) per op, worse cache behaviour) |
| 20 | (v2 PoC, ch. 10) Tier-1 store: an embedded KV (sled), single-node, in wayhouse-controller. Each accepted config submission gets a monotonic revision key; subscribers get a full snapshot + cursor, then a change stream over the same store. No Raft/etcd for this first release — the controller is a single standalone node with no HA yet (see roadmap “PoC” note). | a proper KV beats pushing config files around (crash-consistent, queryable by revision, no ad hoc file-locking) while staying a pure Rust, zero-ops embedded dep — appropriate for a single-node PoC; swapping the backing store for a distributed one (etcd, or Raft over the same KV) is an additive later-release change behind the same store trait, not a rewrite | etcd/Consul (an extra service to run for a PoC that is explicitly single-node); a flat versioned JSON file (no crash-consistent multi-key transactions, reinvents what a KV already gives); embedding Raft now (HA is explicitly deferred past this release) |
| 21 | (phase 12 slice 6, built — wayhouse-controller only; the aggregator’s HA design further down stays design-only) Intra-tier controller HA: embedded openraft 0.9, one Raft group per tier replicating both the config and intent logs together, sled unchanged as the state machine. Writes propose a Raft entry and only commit on the leader; a non-leader replica transparently HTTP-forwards a write to the current leader rather than redirecting. Reads are served by whichever replica got the request, straight from its local sled — no ReadIndex linearizable-read protocol. Cluster membership is static at bootstrap (--ha-peers, identical on every replica — each calls raft.initialize(), a harmless no-op on every node but whichever wins the race); dynamic membership change is deferred. A slave tier’s upward relay runs leader-only, with its cursor (the highest parent revision absorbed) replicated beside each log (built 2026-10-05). Raft RPCs ride the same axum server on a /raft/* prefix gated by a new peer-only --ha-token. | continues ADR 20’s “embed, don’t run an extra service” story instead of trading it away the moment HA enters the picture — etcd would add real operational machinery for a component whose write volume is human-paced, not a hot path; a hand-rolled Raft would reimplement exactly the subtlety (log matching, leader lease safety, snapshotting) a maintained, audited crate already gets right for a security-sensitive component; relaxed reads are justified by the same “freeze on last-known-good” principle (4) the whole chapter already leans on — a follower a few commits behind is the same acceptable staleness as a subscriber on its last replica during an outage, never a divergent one, since sled’s local apply only ever appends; transparent forwarding keeps every existing client (wayhouse, wayhouse-ui, curl) ignorant that HA exists at all | etcd as the Tier-1 backing store (extra service, operational drift from ADR 20’s own reasoning); a hand-rolled Raft/Paxos; two independent Raft groups (config, intent) per tier (two leader elections, two peer lists, possible disagreement about who leads); an HTTP redirect to the leader instead of a transparent forward (pushes leader-awareness onto every client); ReadIndex linearizable reads (real cost for no benefit this design already doesn’t need); dynamic membership on day one (real complexity deferred until a deployment actually needs to resize a live group) |
| 22 | (phase 12 slice 7, built) Staged / canary config rollout: one monotonic revision log (never forked), each revision tagged in a separate stage sled tree with {promoted, canary_groups}. An instance self-reports an optional canary_group when subscribing; “current for group G” = the highest revision that is promoted or has G in canary_groups, computed by an O(revisions) backward scan rather than a maintained index. POST /config/promote/{revision} flips promoted in place — the one deliberate exception to “every change is a new revision,” since promotion changes visibility of existing content, not the content itself. Canary staging applies to config only, never the intent log. | one log avoids ever having to reconcile two branching histories later, which a forked-log design would eventually force; self-reported group membership matches the trust level every other self-reported fact in this control plane already has (IngestPayload::admin_url, an instance’s own --role); an O(revisions) scan is proportionate to a human-paced, hundreds-to-low-thousands-of-entries log — the same “don’t build for a scale problem that doesn’t exist yet” bias the rest of this codebase already uses; intent ops are already narrowly single-scoped (one backend), so “stage this whole document to a subset” has no equally clean meaning for “stage this one op” | a forked/branching revision history per canary group (real merge/reconciliation problem, for a feature meant to be simple); a controller-verified (not self-reported) group membership (real feature, not needed yet, ties into the still-open region-scoped-intent question); an index-backed “current per group” lookup (premature at this scale); extending canary staging to the intent log (no clean partial-rollout semantics for a single op) |
| 23 | (phase 12 slice 8, built) RBAC lives in wayhouse-ui (the only human-facing process), not the controller/aggregator. --users-file (username + argon2 password hash + one of three roles — viewer/operator/admin — matching the API surface’s existing verb tiers) replaces --ui-password for multi-operator deployments; --ui-password is kept as a legacy single-shared-secret (implicitly admin) mode rather than removed. SessionStore maps a session id to a Role; a new require_role(min) middleware layers per route group next to require_session. Audit: wayhouse-ui adds X-Actor: <username> to every write it proxies; wayhouse-controller records it per revision in a new actors sled tree (GET /config/revisions gains an actor field) — a durable per-revision attribution, not a separate durable audit log. | keeps the phase 10+11 slice-11 divergence from docs/10’s original “RBAC lives on the controller” text consistent, rather than half-reverting it — the controller/aggregator still gate on one shared machine token each, exactly as built; argon2 is the currently-conventional choice for this exact job, the same “defer to a maintained crate for a security-critical, well-solved problem” reasoning ADR 16 already used for sniffer sandboxing; three roles match every route in the existing surface without inventing a permission matrix nothing here needs yet; per-revision actor reuses the controller’s already-durable revision history instead of standing up a second audit-log store for a three-role, single-human-facing-process model | RBAC enforced on the controller/aggregator (couples authz to the machine-token model those two intentionally kept simple); a general permission-matrix / policy-engine model (solves a problem this API surface’s three natural tiers don’t have); forcing every existing single-operator deployment onto --users-file (pure friction, no correctness upside below two identities); a separate durable audit-log service (disproportionate to this model’s size) |
| 24 | (phase 13, built — 2026-09-05) Tier-2 regional health fabric: membership via embedded foca (SWIM), per-backend health as a last-writer-wins register piggybacked on foca’s own broadcast/anti-entropy, HMAC-SHA256-authenticated over a per-domain pre-shared key (not mTLS). New wayhouse-core::gossip module (control-plane, spawned only when settings.gossip is set); Backend gains one additive domain_down: AtomicBool next to the existing healthy flag — is_healthy() becomes healthy && !domain_down. Local rise/fall streaks are still the only thing that can set healthy; the gossip module can only ever set domain_down, never clear healthy directly, matching “return to healthy only on this instance’s own say-so.” postcard (new dep) encodes the wire format. | SWIM’s failure-detector subtlety (indirect probing, suspicion timeouts, incarnation numbers) is exactly the “a maintained, audited crate already gets this right” case ADR 21 already established for Raft — reimplementing it by hand is a correctness risk for a security/availability-relevant component, not a simplification; foca’s own broadcast invalidation already gives eventual convergence for a payload this small, so no second anti-entropy framework is needed on top; a shared-secret HMAC is proportionate to an advisory-only signal that can never independently move traffic (only nudge a decision local checks already gate) — full mTLS’s certificate lifecycle is proportionate to Tier 1, which actually accepts writes, not to this | hand-rolled SWIM (reimplements exactly the subtlety a maintained crate already gets right); etcd/Consul-backed membership (a real extra service for an ephemeral, advisory, deliberately-AP signal — contradicts both ADRs 20/21’s “embed, don’t add a service” thread and Tier 2’s own AP design); full mTLS mesh (real cert provisioning burden disproportionate to a signal that can only nudge, never act); a general CRDT-sync/anti-entropy library on top of foca (duplicates what foca’s broadcast handler already does for this payload size) |
| 25 | (phase 14, built — docs/11) Backend transport for globally-distributed proxies: unmodified WireGuard via defguard/wireguard-rs (unifies kernel-netlink and boringtun-userspace config behind one API) as the tunnel data plane — one shared interface with many peers, not one per origin — with a new origin-side wayhouse-agent crate and a new “backend peers” registry on the existing Tier-1 wayhouse-controller for automated key/peer distribution. An origin is modeled as a new BackendSource (ADR 12a) — backend_sources[].type: tunnel — not a new schema concept. The tunnel protocol/crypto is 100% reused, not reimplemented; the only new code is the origin-side interface manager, the control-plane distribution registry, and one new discovery-source implementation. connect_backend/connect_upstream/health.rs’s dials need no changes — a tunneled backend is just a routable SocketAddr on the shared interface’s subnet. | continues the ADR 21/24 “defer to a maintained, audited implementation for a subtle, security-relevant mechanism” reasoning a third time; WireGuard already supports the exact NAT shape this needs (only the proxy side needs a public endpoint; the origin dials out once and roaming + keepalive hold the mapping) with zero new protocol work; a fleet-scale peer registry is the one genuinely missing piece, and it fits the same distribution shape (subscribe + reconcile) ADR 13’s config distribution already established; defguard/wireguard-rs over a hand-rolled kernel/userspace split continues this project’s own “prefer a maintained wrapper over hand-rolled plumbing” bias (ADR 10/12’s nix usage); modeling an origin as a BackendSource reuses ADR 12a’s seam exactly rather than growing a second, parallel discovery concept | a hand-rolled tunnel/framing protocol (reimplements exactly the subtlety a maintained protocol already gets right); adopting rathole/frp directly (standalone tools with their own protocol, no fleet/health/pool awareness, not embeddable); adopting Headscale wholesale (means running real Tailscale clients or reimplementing its coordination protocol — bigger than the one registry actually needed); a bespoke QUIC-based tunnel (real protocol work for no benefit over reusing WireGuard); wireguard-uapi + boringtun hand-joined separately (two APIs to reconcile instead of one); one WireGuard interface per origin (N routing tables/subnets for isolation AllowedIPs already gives per peer); a new origins:/pools[].origin schema concept (duplicates what backend_sources already does) |
| 26 | (built — 2026-10-02) Custom CA trust via --ca-file <PEM> on every binary, additive to the compiled-in Mozilla roots, applied through one shared reqwest-only crate wayhouse-http that holds the extra roots as a set-once process global. wayhouse_http::{builder, client} replace reqwest::Client::{builder, new} at every production call site; each call still builds a fresh client, so no connection-pool or timeout semantics change. The file is fully validated at startup (readable, ≥1 certificate, rustls accepts it). | works in distroless images with no system store; explicit and testable (a fleet test runs a real binary through a private-CA TLS terminator); additive so nothing that works today can break; a set-once global keeps the ~20 call-site diffs to one token each instead of threading a certificate list through every signature; wayhouse-http depends only on reqwest, so wayhouse-aggregator/wayhouse-ui stay free of the data-plane crates | rustls-tls-native-roots (would parse the whole OS store on every Client::new(), and several call sites build a client per request/reconnect; changes trust for every deployment implicitly — can still be added later behind wayhouse-http); threading certificates explicitly through every call path (large signature churn for no behavioural gain); replacing the built-in roots instead of adding to them (breaks public endpoints for anyone who sets the flag) |
| 27 | (built — 2026-10-02; wayhouse-aggregator, wayhouse-ui and wayhouse’s admin API (settings.admin.tls) too, via the shared wayhouse_http::tls::{TlsArgs, serve}) Native TLS for wayhouse-controller (--tls-cert/--tls-key) via a custom axum::serve::Listener in wayhouse-http (wayhouse_http::tls), rustls (ring) with a hot-swappable certificate resolver that re-reads the PEM files when their mtime changes (polled every 30 s); TLS handshakes run in per-connection tasks with a 10 s timeout. A broken replacement keeps the current certificate; bad files at startup are fatal. | no new third-party crate (tokio-rustls/rustls/arc-swap were already in the tree) and every binary keeps the same axum::serve(listener, app) shape; handshakes outside accept() avoid head-of-line blocking by a stalled client; mtime polling renews certbot/cert-manager certificates without a restart or a SIGHUP path the controller doesn’t have; the shared module makes the aggregator/UI/admin API one flag each later | axum-server’s rustls support (a second serving API and dependency tree for one feature); the handshake inside Listener::accept (one slow client stalls every accept); restart-to-renew (an outage per renewal); inotify-based reload (platform-specific, and a 30 s poll is plenty for certificate lifetimes) |
| 28 | (built — 2026-10-02, owner’s request) Security scanning in CI is informational: a trivy job (Trivy over the five deploy/ images from docker save tarballs, plus Cargo.lock and the UI’s package-lock.json; HIGH/CRITICAL with a fix) and an audit job (cargo audit over every Cargo.lock, every push/PR + nightly). Both continue-on-error; results on the run summary, as warning annotations and as artifacts (JSON, SARIF). Trivy runs from the official aquasec/trivy image pinned by digest as a docker:// step; cargo-audit comes prebuilt via the pinned taiki-e/install-action. | the owner wants visibility, not a gate (a new CVE in the distroless base shouldn’t block unrelated PRs); a release binary has no embedded crate list, so the image scan can’t see Rust crates — the lockfile scans stand in, and cargo audit adds the RustSec-only advisories Trivy (GHSA only) misses; a digest-pinned image whose binary was checked byte-identical to the checksum-verified release can’t be swapped later; a separate job keeps the scans’ outcome visible as its own pipeline stage | aquasecurity/trivy-action / setup-trivy (their tags were hijacked in March 2026 and a backdoored Trivy 0.69.4 published); a job-level container: on the Trivy image (GitHub can’t run JavaScript actions in the Alpine image); blocking scans; GitHub’s Security tab via SARIF upload (a private repo needs GitHub Code Security — a follow-up); one job per image (the ~120 MB vulnerability DB would be fetched five times) |
| 29 | (built — 2026-10-03) TLS handshake flood limits in wayhouse_http::tls::TlsListener: a 3 s ClientHello deadline (via LazyConfigAcceptor, inside the 10 s handshake timeout), at most 16 pending handshakes per source (IPv4 address / IPv6 /64, closed at once over it), and a global cap of 512 that admits the new connection and aborts the oldest pending handshake. Fixed constants (HandshakeLimits), no flags. | bounds tasks and fds under a many-source flood without the lockout a refuse-at-cap global limit has (an attacker must open 512 connects within a real client’s one-RTT handshake to evict it); per-source /64 because one IPv6 host owns a /64; the short ClientHello deadline makes an idle connect cost 3 s, not 10 | no cap (a many-source flood bounded only by the fd limit, where EMFILE locks everyone out); a global cap that refuses at the limit (~cap idle connects lock every client out — tried and dropped 2026-10-02); a per-source connection rate limit (more state, and the pending cap already bounds what a source holds); flags per binary (no operator has asked; HandshakeLimits keeps that a small change) |
| 30 | (retrospective record of ADR 20/21, written 2026-10-04) sled 0.34 stays the controller’s embedded store (config/intent logs, stage, actors and address trees, and the openraft log and state machine), accepted as a known dependency risk, with no migration planned until the HA log store is next reworked. sled is used through a narrow surface: open, open_tree, ordered keys and range/prefix scans, flush/flush_async, and multi-tree transactions, all confined to wayhouse-controller. The risks are real and accepted knowingly. The stable line, 0.34.7, was published 2021-09-12, and the only newer releases are 1.0.0-alpha.* (the latest, alpha.124, is from 2024-10-11), so the crate has been pre-1.0 and effectively unmaintained for years; it is behind the fxhash and instant unmaintained-crate warnings from cargo audit (informational; they do not fail the target); the on-disk format is not guaranteed stable across the 1.0 rewrite, so moving to 1.x is a migration, not a bump; and a crash or bug in the store would hit the Raft log, which is the worst place for it. Mitigations already in place: sled is used only inside wayhouse-controller (eight source files, the main seams being store.rs and ha/log_store.rs), so a swap is bounded to one crate; Raft replication means a damaged replica can be rebuilt from a peer’s snapshot (ha_snapshots test); the file-lock release race is handled with a retry on reopen (HANDOVER, known flakes); and write volume is human-paced, not a hot path. Revisit when the HA log store is next changed, an advisory lands against sled itself, or a data-loss bug is reproduced. At that point evaluate redb first, then fjall (see alternatives); do not migrate opportunistically. | a migration now would touch the config store, the Raft log store and the state machine at once, and buys nothing a user can see; the advisories are informational; the surface used is small and confined to one crate, so a later swap is bounded work; pure Rust and zero-ops (ADR 20) still hold; a rewrite of the controller store is better done together with a deliberate HA-log change than as drive-by churn | redb (pure Rust, copy-on-write B-tree, ACID, stable file format, 4.x and actively released as of 2026-10-02; the leading candidate: it has multi-table write transactions and range scans, but no built-in watch/subscribe and its single-writer model needs the change-stream logic re-expressed on top); fjall (pure Rust LSM-tree, 3.x and actively released as of 2026-10-03; cross-keyspace write batches and prefix/range iteration, a good fit for an append-heavy log, but a larger and younger dependency tree, and compaction and background threads add operational behaviour to reason about); sled 1.0 alpha (the maintained line of the same crate, but still alpha with a new on-disk format and no stability promise, so it trades one risk for a bigger one); SQLite via rusqlite (very mature and well understood, but a C dependency, which cuts against the pure-Rust, zero-ops choice of ADR 20); RocksDB (mature, but a large C++ build and a heavy dependency for a human-paced log); a hand-written append-only log plus snapshots (full control, but reimplements crash consistency, the exact subtlety ADR 21 chose to delegate) |
| 31 | (perf pass, built — 2026-10-05, closes #51) UDP egress batches with sendmmsg(2) on both legs. The reply pump pulls whatever is queued on its connected upstream socket with one recvmmsg into a per-thread ReplyBatch (16 × 64 KiB, ~1 MiB per worker thread, shared by every session on that thread, never held across an .await) and hands it to the client with one sendmmsg (one address, and in prefix mode one shared pktinfo source cmsg, for the whole batch). The ingress loop groups consecutive datagrams of one (client, dst) flow that hit an existing session and forwards the run with one sendmmsg on the session’s connected socket. A partial send resumes at the first unsent datagram; a full client socket parks a right-sized copy of only the unsent tail and awaits writability. Non-Linux keeps one datagram per syscall behind the same send_batch / ReplyBatch API. | the RSS objection in #51 (a RECV_BATCH × MAX_DATAGRAM buffer per session would be 16× the RSS) does not apply since #122 moved reply buffers to a per-thread scratch, so the batch costs ~1 MiB per thread, not per session; replies of one session share the destination so sendmmsg fits without per-datagram allocation; a connected upstream socket needs no per-message address | per-session reply batches (16× RSS); per-datagram allocation; io_uring (bigger dep); batching only the reply leg (the forward leg is the same win for client→server game traffic) |