Documentation

Note

wayhouse is a work in progress. These documents describe the design the code implements, but details can still change. See the notice in the top-level README.

Design documents (00–12)

The numbered chapters are the source of truth for the design: when the code and a chapter disagree, the chapter wins and the code (or the chapter) gets fixed.

Start here: 00 Overview for the problem and the vocabulary, then 02 Architecture for how the pieces fit together.

Feature specs and plans

superpowers/ holds the design spec and the step-by-step implementation plan written before each larger change since the roadmap phases were finished. They are kept as a record of why things were built the way they were; see superpowers/README.md.

Elsewhere in the repository

• ../deploy/README.md: reference images, Compose demo and Kubernetes manifests.
• ../crates/wayhouse-sniffer-abi/README.md: the WASM sniffer ABI (guest side); the official sniffers are in wayhouse-proxy/sniffers.
• upgrading.md: upgrading a fleet one component at a time: compatibility window, version table, order of operations, rollback.
• web-proxy.md: publishing HTTP(S) sites through the packaged Caddy (wayhouse-web-sync, the wayhouse-caddy image, compose and Kubernetes recipes).
• sniffers.md: sniffer manifest, registry index and verification formats.
• ../crates/wayhouse-bench/README.md: the latency / load harness.
• ../crates/wayhouse-config/fuzz/README.md: fuzz targets.
• ../HANDOVER.md: current state, known follow-ups and gotchas.

00 – Overview

Problem

Exposing game servers directly to the internet creates recurring problems:

• The real server IP is public and therefore directly exposed to DDoS.
• Many server instances (matches, lobbies, regions) require many ports/IPs and some external component that steers clients to the right instance.
• Restarts/deployments disconnect active players.
• Operations (metrics, connection logs, block lists) have to be rebuilt per game.

Existing solutions like BungeeCord/Velocity (Minecraft) or game-specific master servers each solve this for one game only, and often for one transport protocol only.

Goal

A reverse proxy for game servers that works protocol-agnostically at layer 4 (TCP + UDP) and optionally derives routing decisions from the first handshake packet via pluggable L7 sniffers. The core knows no game; game knowledge lives exclusively in optional sniffers or in external routing logic.

Use cases

1. IP protection / edge termination – clients connect only to the proxy; backends sit in a private network. The proxy is the single DDoS-exposed point.
2. Fan-out to many instances – one public endpoint play.example.com:7777 distributes to hundreds of match containers.
3. Region/hostname routing – eu.example.com and us.example.com on the same port, selected by SNI (TLS) or by the first packet.
4. Matchmaker integration – on connect, the proxy asks an external service (gRPC/HTTP): “Where does this player / this token belong?”
5. Zero-downtime deployments – a new backend version is registered, the old one is set to “draining”; existing sessions drain, new ones go to the new version.
6. Blue/green & canary for game server builds.
7. Central operational view – uniform metrics (connections, throughput, added RTT, errors) and connection logs across all games.
8. Access control – IP allow/deny lists, geo filters, rate limiting, per-backend connection caps.
9. Fleet management (v2) – one control plane + web UI to view and configure many proxy instances across regions from a single authoritative place, with revision history and persisted operator intent (see 10-distributed-control-plane.md).

Non-goals

• No game logic, no matchmaking, no anti-cheat, no lobby system.
• No parsing/rewriting of game packets in the core (only optional sniffers may read the first N bytes — never modify the running stream).
• Not a NAT replacement for arbitrary client-to-client traffic (no TURN/relay mesh).
• Does not force transport encryption where the game does not use it (optional TLS/DTLS wrapping is an add-on feature, not a requirement).
• Not a replacement for an L3 scrubbing provider against volumetric attacks — the proxy complements it, it does not replace it.
• No cross-instance session state. Instances are independent on the data path; HA is an anycast / L4-LB concern. The v2 control plane (10) shares config and health, never sessions — a client rehashed to another instance reconnects.

Guiding principles

• Latency first. Every extra hop costs RTT. The data path must be allocation-light, copy-light, and free of unnecessary serialization.
• Agnostic by default, sniffer as the exception. Pure L4 forwarding must work without any sniffer.
• Separate control plane from data plane. Config/backend changes must not block the data path.
• Operations is a feature. Health checks, draining, metrics, reload belong in the first version, not in “later”.

Glossary

01 – Requirements

Functional requirements

F1 – Listeners

• F1.1 Multiple listeners at the same time (different IP:port/protocol).
• F1.2 Transport: TCP and UDP; QUIC/DTLS as “opaque UDP” with no special handling.
• F1.3 Dual stack (IPv4 + IPv6).
• F1.4 Optional port-range listener (one listener for 30000–30999) for games with dynamic ports and for subdomain-per-port routing (scheme B).
• F1.5 Optional prefix listener: bind to a routed IP prefix with one socket, destination address per packet/connection via IP_PKTINFO/getsockname. Basis for dst routing for games without a protocol hint (scheme A).

F2 – Routing

• F2.1 Static listener → pool mapping.
• F2.2 Routing by SNI (on a TLS handshake) without terminating TLS.
• F2.3 Routing by first-packet match (byte prefix, regex over the first N bytes).
• F2.4 Routing via an external resolver (gRPC/HTTP callback) with cache & timeout fallback.
• F2.5 Routing by client IP / geo (CIDR lists).
• F2.6 Routing by destination IP/prefix (dst) and destination port — for games that send raw data to a fixed IP:port with no hostname/SNI/token. The subdomain is mapped by DNS to its own destination IP (scheme A) or its own port (scheme B).
• F2.7 Push resolver: short-lived src_ip → pool mapping set in advance by a launcher/API (scheme C).
• F2.8 Fallback/default route when no rule matches.
• F2.9 Deterministic rule priority (first matching rule wins).

F3 – Upstream / pool

• F3.1 Selection strategies: round-robin, least-connections, consistent hashing (on client IP or routing key), weighted, “first available”.
• F3.2 Session affinity: same 4-tuple / same routing key → same backend, as long as that backend is healthy.
• F3.3 Health checks: active (TCP connect, UDP probe, HTTP GET on a sidecar port) and passive (error rate/timeouts observed in the data path).
• F3.4 Backend states: healthy, unhealthy, draining, disabled.
• F3.5 Max sessions / max new sessions per second per backend.

F4 – Session management

• F4.1 UDP session table keyed by 4-tuple with a configurable idle timeout.
• F4.2 TCP: normal connection lifetime, half-close propagation.
• F4.3 Graceful draining: redirect new sessions, end old ones after a grace period.
• F4.4 Caps: global and per-listener connection limits.

F5 – Client-IP preservation (details in 04)

• F5.1 PROXY protocol v1/v2 for TCP, optional per pool.
• F5.2 PROXY protocol v2 for UDP (header in the first datagram) as an option.
• F5.3 Transparent mode (TPROXY, IP_TRANSPARENT) on Linux.
• F5.4 Without any of that: only the proxy IP is visible (documented behavior).

F6 – Configuration & control plane

• F6.1 Declarative file (YAML) as the source of truth.
• F6.2 Hot reload without dropping connections (SIGHUP / file watch / API).
• F6.3 Admin API (HTTP/gRPC) for: register/deregister backend, set state, read config snapshot, trigger draining.
• F6.4 Optional dynamic backend discovery adapter (DNS SRV, Consul, Kubernetes EndpointSlices) behind a stable internal interface.
• F6.5 Validation on load; an invalid config is rejected and the old one stays active.

F7 – Observability (details in 06)

• F7.1 Prometheus metrics.
• F7.2 Structured connection logs (JSON), sampleable.
• F7.3 Optional OpenTelemetry tracing for the connection-setup phase.
• F7.4 Health/readiness endpoints for the proxy itself.

F8 – Security (details in 07)

• F8.1 IP allow/deny lists (CIDR), per listener.
• F8.2 Rate limiting: new connections/datagrams per source IP and per subnet.
• F8.3 SYN-flood protection (SYN cookies via the kernel), UDP amplification protection (respond-only-after-first-valid-packet heuristic, optional sniffer).
• F8.4 Resource caps against memory/FD exhaustion.

Non-functional requirements

Also: horizontally scalable (stateless w.r.t. persistent storage, session state local only), Linux as the primary platform, container-friendly, runs without root after setup (capabilities instead of root).

Assumptions

• Backends are reachable over a trusted internal network. Superseded by phase 14 (backend transport, built): see 11-backend-transport.md for how a proxy reaches a backend that isn’t on its local network (a WireGuard tunnel per origin). A same-network deployment needs none of that and this assumption still holds for it.
• The client reaches the proxy via DNS; the proxy does not need to cryptographically verify client identity (the game does that if needed).
• Volumetric L3/4 attacks are absorbed upstream (scrubbing/anycast).

Open questions

• Must the proxy serve multiple backends at once per client (e.g. TCP control + UDP gameplay on separate instances)? → affects the session model.
• Do we need cross-instance session sharing (failover of a live UDP session to a different proxy instance)? First version: no.
• Is QUIC-aware routing (connection ID) required, or is opaque UDP enough?

02 – Architecture

Overview

One proxy instance in context. Traffic enters on the left and leaves on the right; sniffers run inside the proxy on the data path, while plugins run in the controller, off it.

Blue boxes are the data path (ingress left, egress right); amber boxes are everything off it. Dotted lines are control, config or state flows.

The immutable snapshot is not tied to one component: it carries the listener definitions with their filters and routes, the pools with their backends and health, and the resolver and discovery settings, so the diagram points it at each of them.

Data plane / control plane separation

• Data plane: accepts connections, makes the routing decision against an immutable config snapshot, pumps bytes. No lock on the hot path; snapshot swap via an atomic pointer swap (arc-swap pattern).
• Control plane: builds new snapshots (from file + discovery + admin API), runs health checks, exports metrics/logs. Runs in its own tasks/threads, may block.

A snapshot contains: listener definitions, the compiled routing table (including precompiled matchers), pools with their current backend list and health state. Health updates produce a new snapshot (cheap copy-on-write of the affected pool struct, the rest is shared).

Components

1. Listener

• Binds socket(s). TCP: accept loop or SO_REUSEPORT sharding across workers.
• UDP: one/more recvmmsg loops per socket, SO_REUSEPORT for scaling.
• Applies early filters: IP allow/deny, connection rate limit (cheap, before any allocation).
• Hands off to the router: for TCP after an optional peek of the first bytes; for UDP with the first datagram.

2. Router

• Selects the first matching route (deterministic priority).
• Matcher types: always, sni, first-bytes (prefix/regex/length), client-cidr, dst (destination IP/prefix), port (destination port), external.
• Determine destination address: with a prefix bind (one socket for a whole prefix), via IP_PKTINFO/IPV6_RECVPKTINFO (UDP, per datagram) or getsockname() (TCP). This is the lever for games with no protocol hint: subdomain → its own destination IP.
• First-packet peek (TCP): MSG_PEEK up to peek_max_bytes or peek_timeout. If that is not enough (the client sends nothing first), the default route applies.
• External resolver: calls gRPC/HTTP with {listener, src_ip, sni, first_bytes(b64), routing_key}, expects {pool | target, ttl}. The result is cached (key configurable). Timeout → fallback route.
• Result: a target pool or a fixed target.

3. Upstream / pool

• Holds the backend list + states from the snapshot.
• The LB strategy picks a healthy backend; a consistent_hash pool picks by client IP (or IP and port), stateless.
• Checks per-backend caps (max sessions, new-session rate). No candidate → defined error handling (TCP: RST/close; UDP: drop datagram + metric).

4. Connection / session pump

• TCP: bidirectional copy. Linux fast path via splice() between the two sockets (zero-copy), fallback to user-space buffers. Propagate half-close, timeouts (idle, connect), pass through TCP_NODELAY.
• UDP: one connected upstream socket per session (connect(2)) so replies find their way back without a table lookup. Session table: HashMap<4-tuple, Session> + an idle-timer timing wheel. The idle timeout closes the session.
• PROXY protocol: when the option is enabled, write the header before the first payload to the backend (TCP) or prepend it (UDP, first datagram).

5. Health checker (control plane)

• Active checks per backend by interval/timeout/thresholds (rise/fall).
• Passive signals from the data path (connect errors, early resets, UDP “port unreachable” / ICMP) feed in as events.
• Publishes a new snapshot on a state change.

6. Config loader & discovery adapter

• Loads/validates YAML, builds the snapshot, compiles matchers.
• Adapters (DNS SRV, Consul, K8s EndpointSlices, static) implement a common BackendSource interface and provide backend lists per pool.
• Reload: new snapshot becomes active atomically; listeners are reconciled by name (added spawned, removed stopped, changed re-spawned) — an unchanged listener keeps running.

7. Admin API

• GET /config (active snapshot, plaintext — implemented, phase 5), GET /pools, GET /sessions[?listener=&pool=&proto=&src=] (live connection / session list — implemented)
• POST /admin/drain / POST /admin/undrain (flip readyz for the LB — implemented, phase 5)
• PATCH /pools/{p}/backends/{addr} {state: enabled|draining|disabled}, POST /pools/{p}/backends {addr}, DELETE /pools/{p}/backends/{addr} — all implemented (phase 5). Add/remove edits live in a runtime overlay layered on the file config, so they survive a file reload.
• POST /route-hint – push resolver: {src_ip, pool|target, ttl_sec} for games with no protocol hint, set by launcher/matchmaker (scheme C in 03)
• POST /reload, GET /healthz, GET /readyz, GET /metrics
• Auth: mTLS or bearer token, bound to an internal interface only.

Threading / runtime model

• One worker per CPU core, SO_REUSEPORT sockets per worker → no accept contention.
• Each session is pinned to its accepting worker (thread-local session table → no locks in the UDP path).
• Control-plane tasks (health checks, reload, discovery refresh, admin API) run as ordinary tasks on the same shared multi-thread tokio runtime as the data plane, not a separate pool — simpler than the originally-planned split, and safe because control-plane work never blocks (rule: it may be slow, but never blocking-syscall slow) and never touches the hot path’s locks.
• One dedicated OS thread (not a tokio task) for the sniffer engine’s epoch ticker — a wasmtime::Engine::increment_epoch heartbeat that arms the per-call timeout for WASM sniffers. Spawned once with the SnifferLoader, runs for the process lifetime, does no I/O. Absent when no settings.sniffers is configured, and absent entirely from a build without the wasm-sniffers cargo feature (which refuses a settings.sniffers block at startup).
• Memory: per-worker pre-reserved buffer pools; session structs from a slab allocator to avoid fragmentation.

Failure & error behavior

• Backend fails during a session: TCP → close both sides, metric; UDP → the session stays briefly (configurable), then dropped. Optionally “rehome” new datagrams to another backend if no affinity is required.
• Proxy instance fails: clients reconnect; an L4 LB / anycast in front of the proxy spreads them onto the remaining instances. No shared session state (v1, and no shared session state in v2 either — see below).
• Overload: early rate limits + accept throttling + load shedding with a metric, before the latency of existing sessions suffers.

Extension points

• Sniffer API: fn sniff(&[u8]) -> Option<RouteHint> (SNI, Minecraft, Steam A2S, FiveM …). SNI is built in; the others are sandboxed WASM modules loaded from settings.sniffers.dir (see sniffers.md). They run on the data path and only see the bytes they are handed.
• Plugins are a separate mechanism: WASM modules hosted by the controller, off the data path, reacting to timers, webhooks and events through capabilities the operator approved (see plugins.md). They can publish routes, which proxies merge after their own.
• BackendSource adapters (see above).
• Filter chain before routing (ACL, rate limit, geo) as an ordered, configurable list.

Multi-instance & the distributed control plane (v2)

The diagram above is one instance. Production runs many identical, independent instances across hosts / AZs / regions, fronted by anycast or an L4 LB; they share nothing on the data path (ADR 4).

10-distributed-control-plane.md plans the v2 additions that give a fleet operator (and a management GUI) one authoritative config, persisted operator intent, and multi-vantage health — without touching the hot path. In brief:

• Tier 1 — a global, ordered, durable store for structural config and operator intent (the phase-5 overlay, admin state, route hints). Instances pull revisions and feed them through the same validate() → Snapshot::build → ArcSwap::store path a file reload uses. Written by a new optional wayhouse-controller, which also aggregates fleet reads and hosts the GUI + auth.
• Tier 2 — a per-failure-domain gossip fabric for observed backend reachability. Advisory and rebuildable: each instance’s own checks still decide, with the domain quorum as weighted input.
• Unchanged: the data plane, the immutable-snapshot invariant, and “no shared session state”. Rate-limit buckets and UDP session tables stay purely instance-local.

Maps onto the predicate the data plane already computes: takes_new_sessions() == is_healthy() && admin_state() == Enabled — Tier 2 feeds the left half, Tier 1 the right.

03 – Routing

Goal: map an incoming connection/session to a pool (and optionally an affinity key) — ideally without game-protocol knowledge, with optional sniffers where needed.

Implementation status (phase 3, partial). listeners[].routes is a priority-ordered [{ match, action }] list, first match wins, with action: { pool: <name> }. Implemented matchers: always, client_cidr (source IP), dst (destination IP the client connected to, via getsockname / the bind address, against a cidrs list — see the caveat below), port (destination port from the accepting socket), first-bytes (a prefix — "hex:..." | "ascii:...", ≤ 512 B — and/or a length: { min, max } byte-count window; on TCP length sees only what one peek returned, on UDP the exact datagram length), and sni (host patterns: exact, *.suffix, .suffix — matched against server_name from the peeked, non-terminated TLS ClientHello; TCP listeners only). The sniffer matcher exists ({ type: sniffer, sniffer: <name>, host: [...] }, several per listener) but no sniffers are built in — a sniffer: route never matches until a sniffer is loaded (Phase 9). The TCP path MSG_PEEKs up to 4096 B (250 ms budget) before routing, only when a route needs bytes; UDP inspects the first datagram it already holds. A ClientHello split across TCP segments is reassembled: the path re-peeks every 5 ms until the whole first TLS record is buffered or the 250 ms budget expires, so sni still matches a multi-segment ClientHello. (first-bytes length on TCP is still “what has been peeked so far” — a prefix fits in the first segment.) The consistent_hash balancer is implemented (balancer: consistent_hash, pool-level hash_on: src_ip | src_ip_port; rendezvous/HRW hash over the healthy backends); so is weighted (balancer: weighted + a weights: map). dst — two forms: a normally-bound TCP/UDP listener sees only its own bind IP (useful across addresses the host serves separately; TCP can bind a non-local address with freebind: true). A UDP listener with prefix: <cidr> runs in prefix mode — one wildcard socket with IP_PKTINFO / IPV6_RECVPKTINFO serves the whole routed prefix (scheme A below): the real per-datagram destination feeds dst, replies go out from that address, and datagrams outside the prefix are dropped (wayhouse_datagrams_dropped_total{reason="outside_prefix"}). The sniffer seam is wayhouse_core::sniff::Sniffer → RouteHint { host, key, reject }, fed into routing before matchers run — a reject hint drops the connection / datagram outright (result="sniffer_reject" / reason="sniffer_reject"); loading real sniffers (sandboxed, from a separate repo) is Phase 9. The push resolver (POST /route-hint) is implemented: a listener with route_hint: true checks a short-lived src_ip → pool table before its route list (wayhouse_route_hints_applied_total{listener} counts hits). The external resolver (action: { resolver: <name> }, resolvers: section) is implemented — phase 4 slices 1–4: HTTP and gRPC transports (type: http | grpc), pool and target results (target = connect straight to a resolver-named instance, no pool / health / cap), on_error: reject | fallback_route | stale_ok, and a TTL’d LRU result cache (cache: { key, positive_ttl_sec, negative_ttl_sec, max_entries }; key parts src_ip / src_ip_port / sni / routing_key / first_bytes:a:b). Only The resolver has no affinity key (sticky_key was removed, #54): affinity comes from consistent_hash pools and the request-keyed cache. Still pending elsewhere: the TCP side of prefix binding beyond freebind and the first_available balancer. The weighted balancer is implemented (balancer: weighted + a weights: map of "ip:port" → share, default 1) — weighted round-robin over the healthy set, one atomic tick per selection. Regex-over-first-bytes is folded into the Phase 9 sniffer layer, not a first-bytes sub-form. A listener with a bare pool: is normalised to one always route.

Evaluation order

1. Early filters (ACL, rate limit, geo) – before routing, may reject immediately.
2. Listener binding – the listener may already map 1:1 to a pool (simplest case, no further logic).
3. Sniffer (if a sniffer route is configured) – runs once on the peeked first bytes. A listener may route on several sniffers (e.g. quic, wireguard and a2s on one UDP port): they are tried in order of first appearance in the route list and the first that recognises the bytes wins — a later sniffer is not consulted, even if none of the winner’s routes match (routing then falls through to non-sniffer routes such as always) — and a sniffer route only matches the sniffer it names. A RouteHint { reject: true } drops the connection / datagram immediately (before the push-resolver hint, so a spoofable src_ip hint cannot override it); TCP wayhouse_listener_connections_total{result="sniffer_reject"}, UDP wayhouse_datagrams_dropped_total{reason="sniffer_reject"} and no reply. A non-reject hint is carried into route matching.
4. Push-resolver hint – if the listener has route_hint: true and a live src_ip → pool entry exists (from POST /route-hint) whose pool still exists, it wins and steps 5–7 are skipped.
5. Route matching – the listener’s ordered rule list; the first matching rule wins. Each rule: match + action (pool or resolver).
6. Resolver (if the rule requires it) – external lookup with cache/fallback.
7. Default route – when nothing matches.

Matcher types

always

Always matches. For default/catch-all rules.

port

The destination port (of the accepting socket) selects the pool; ports: accepts a "lo-hi" range too. Pairs naturally with a port-range listener (bind: "host:lo-hi", F1.4) — one config entry spawning a real socket per port across the range, so 30000–30099 → pool-a needs one listener, not one per port. See 05.

dst (destination IP / prefix of the incoming packet)

The address the client sent to — not from the payload but from the socket. Compared against a CIDR/prefix list (LPM trie). This is the lever for games with no protocol hint at all: the subdomain is mapped by DNS to its own destination IP and the proxy distinguishes on dst. See the section “Routing without a protocol hint”. Data-path requirement: receive on a whole prefix with one socket (IP_PKTINFO / IPV6_RECVPKTINFO), see 04.

client_cidr

Source IP in a CIDR list. Uses: internal testers to a staging pool, region roughly by IP block, partner ranges. Also for the launcher/push resolver (section below): short-lived src_ip → pool mapping.

sni (TCP, TLS handshake – without termination)

• Reads the ClientHello, extracts server_name.
• Comparison exact / suffix (*.eu.example.com) / regex.
• TLS is not terminated; the byte stream (including the ClientHello) is forwarded unchanged. Peek only.
• Cost: one MSG_PEEK, a minimal parser (no OpenSSL required).

first_bytes (TCP or UDP first datagram)

• prefix: exact byte/string prefix (hex: or ascii: notation).
• regex: regex over the first N bytes (precompiled, N bounded).
• length: datagram length within a range (coarse heuristic, e.g. query vs. gameplay).
• sniffer: <name>: a named sniffer returns structured hints, e.g.:
  • sni (generic, see above)
  • minecraft → handshake hostname + protocol version
  • a2s / source-query → Valve query recognized (route to a query pool)
  • quic → QUIC Initial recognized (v1, v2, IETF drafts; key quic). The sniffer also decrypts the Initial (its keys derive from the packet’s own destination connection ID and a public salt, RFC 9001 §5.2) and returns the TLS SNI as the hint’s host (v1, v2 and drafts 29 to 34). When the SNI cannot be read (an older draft, or a ClientHello split across Initial packets with the SNI in a later one) only the key is set, so a host: route does not match and an empty-host: route still does
  • wireguard → handshake initiation recognized (key wireguard)
  • openvpn → client hard reset recognized, UDP or TCP-framed (key openvpn; a weak one-byte signal: about 3 in 256 random datagrams match, so it makes the first-packet gate leaky and belongs after stronger sniffers in the sniffer list)
  • raknet → RakNet offline handshake recognized by its magic (Minecraft Bedrock and other RakNet games; key raknet)
  • teamspeak3 → TeamSpeak 3 TS3INIT1 client init recognized (key teamspeak3)
• Sniffer contract: read-only, receives up to peek_max_bytes, returns Option<RouteHint { key?: String, pool_hint?: String, reject?: bool }>. No access to later bytes, no writing.

external (resolver)

• A rule action instead of a fixed pool.
• Request to a gRPC/HTTP endpoint:

  {
    "listener": "public-udp",
    "src": "203.0.113.7:51343",
    "dst": "198.51.100.10:7777",
    "sni": "eu.example.com",
    "first_bytes_b64": "AAECaGVsbG8=",
    "routing_key": "eu.example.com"
  }
  

• Response:

  { "pool": "match-eu-1", "target": null, "ttl_sec": 30 }
  

  either pool (then normal LB) or target (a fixed instance, e.g. assigned by the matchmaker).
• Cache: key configurable (src_ip, sni, routing_key, hash of the first bytes). Positive TTL from the response, negative TTL separately.
• Timeout/error: on_error: fallback_route | reject | stale_ok.
• Use: the matchmaker assigns a player token to an instance; the proxy does not need to understand the token, only forward it and cache the decision.

Load balancing within a pool

Session affinity

• Implemented: a consistent_hash pool (rendezvous hash of src_ip or src_ip_port). Stateless, so it holds at any worker count and after any idle eviction; an unhealthy or draining backend is skipped and its clients move to the next-highest score. Adding a backend remaps about 1/N of the clients.
• Not implemented: a key → backend_id (+ TTL) sticky table keyed by the sniffer key. The UDP table that existed was removed in #56: it was per worker, so with SO_REUSEPORT (clients spread by source port) it kept only ~34% of clients on 4 workers.
• For UDP, affinity is effectively mandatory (otherwise the gameplay stream fragments across multiple instances): point UDP listeners at a consistent_hash pool.

Routing without a protocol hint (raw data to an IP:port)

Many games send raw, often encrypted payload from the first byte to a fixed IP:port — no hostname, no SNI, nothing stable in the first packet. Then there is nothing in the stream to tell survival from creative.

Principle: route by what you still know without the payload — the destination address (dst_ip) and the destination port the client sent to. So the “subdomain” must be mapped by DNS to a distinguishable (IP, port) combination. Three schemes, used alone or combined:

Scheme A – one IP per server (recommended, especially with IPv6)

• DNS maps the name to its own address:

  survival.example.net   AAAA  2001:db8:ace:1::1
  creative.example.net   AAAA  2001:db8:ace:1::2
  hardcore.example.net    A    198.51.100.7      # IPv4 only when necessary
  

• The proxy is given a routed prefix (e.g. an IPv6 /64 or /48) and accepts traffic on all addresses in it with one single wildcard socket — no socket per IP:
  • UDP: IP_PKTINFO / IPV6_RECVPKTINFO yields the actual destination address per datagram; the reply is sent via cmsg with the same source address.
  • TCP: IP_FREEBIND or net.ipv4.ip_nonlocal_bind / bind-any on the prefix; the destination IP of the accepted connection comes from getsockname().
• Routing table: a dst matcher against an LPM trie prefix → pool. Millions of destination IPs cost almost nothing (memory ~ number of rules, not number of IPs).
• IPv6 is the normal case here: address space is effectively unlimited, each instance can get its own public address. IPv4 only with a small, expensive IP pool — then prefer scheme B.

Scheme B – one port per server

• All names point to the same IP; the port distinguishes them:

  survival.example.net → 198.51.100.7:30001
  creative.example.net → 198.51.100.7:30002
  

• Where the game/protocol supports it, an SRV record hands out host+port (_game._udp.survival.example.net SRV 0 0 30001 edge.example.net). Otherwise the launcher / server-browser config carries the port.
• Proxy: a port-range listener (bind: "0.0.0.0:30000-30099", F1.4) + port matcher (30001 → pool-survival).
• Limits: some clients hard-code the port; restrictive client firewalls; SRV support is rare outside a few protocols.

Scheme C – pre-registration via launcher/API (push resolver)

• If the game starts through a launcher or server browser, that calls the control plane API before connecting: POST /route-hint { src_ip: "203.0.113.7", pool: "survival", ttl_sec: 30 }.
• The proxy keeps a short-lived src_ip → pool table. The first packet/SYN from that IP with no other hint is resolved via it, then the pool’s balancer takes over.
• Weakness: several players behind one NAT IP wanting different subdomains at the same time cannot be told apart — unless the launcher can additionally set a short token that does end up in the first packet (then first-bytes). Otherwise fall back to scheme A/B.
• This is the “reverse” of the external resolver: push instead of pull, useful when the proxy cannot query the client context itself.

What fundamentally does not work

Same game, same IP, same port, no token in the first packet, multiple logical servers behind it: not distinguishable. Then there is no way around scheme A (own IP) or B (own port) — or a launcher that supplies a token.

DNS and routing table from one source

To keep name → (IP/prefix | port) → pool consistent, one inventory file (per server: name, address or port, pool) should generate both the DNS zone and the listeners[].routes with the dst/port match. Otherwise resolution and forwarding drift apart.

Examples

A) One port, many match instances, matchmaker decides

listener public-udp (0.0.0.0:7777/udp)
  route 1: match = always  → action = external(resolver = matchmaker)
     resolver cache key = first_bytes(0..16)   # contains a session token
     on_error = reject

B) Region by SNI, TLS stays with the backend

listener public-tcp (0.0.0.0:443/tcp)
  route 1: sni suffix ".eu.example.com" → pool match-eu   (consistent_hash on sni)
  route 2: sni suffix ".us.example.com" → pool match-us
  route 3: always                        → pool lobby-default

C) Minecraft network (sniffer only reads the hostname)

listener public-tcp (0.0.0.0:25565/tcp)
  route 1: sniffer minecraft, host == "survival.example.net" → pool mc-survival
  route 2: sniffer minecraft, host == "creative.example.net" → pool mc-creative
  route 3: always → pool mc-lobby

D) Separate query and gameplay pools by packet length

listener public-udp (0.0.0.0:27015/udp)
  route 1: first-bytes prefix hex:FFFFFFFF → pool source-query   # A2S
  route 2: always                          → pool gameplay

E) Raw UDP with no hint at all – subdomain by destination IP (scheme A)

DNS:  survival.example.net  AAAA  2001:db8:ace:1::1
      creative.example.net  AAAA  2001:db8:ace:1::2
      arena.example.net     AAAA  2001:db8:ace:1::3

listener raw-udp
  bind:      "[2001:db8:ace:1::]/64 :7777/udp"   # one wildcard socket, IPV6_RECVPKTINFO
  routes:
    route 1: dst 2001:db8:ace:1::1/128 → pool survival
    route 2: dst 2001:db8:ace:1::2/128 → pool creative
    route 3: dst 2001:db8:ace:1::3/128 → pool arena
    route 4: always                     → reject   # unknown destination IP

The client connects to survival.example.net:7777, immediately sends raw packets; the proxy reads the destination address from the datagram cmsg and picks the pool. Replies go back with 2001:db8:ace:1::1 as the source.

F) Like E, but only IPv4 available – subdomain by port (scheme B)

DNS/SRV:  _game._udp.survival.example.net  SRV 0 0 30001 edge.example.net
          _game._udp.creative.example.net  SRV 0 0 30002 edge.example.net

listener raw-udp-v4
  bind:  "0.0.0.0:30000-30099/udp"
  routes:
    route 1: port 30001 → pool survival
    route 2: port 30002 → pool creative
    route 3: always     → reject

(Pseudo-code — real YAML has no /udp suffix on bind, protocol: udp is its own key, and routes: entries are { match: {...}, action: {...} }. See docs/05 for the actual shape; the range-bind idea itself (F1.4) is real and built.)

G) Web sites behind Caddy (web:)

listener https   bind 0.0.0.0:443   route 1: sni *.example.com → pool web   (proxy_protocol: v2)
listener http    bind 0.0.0.0:80    route 1: always            → pool web
pool web         targets [127.0.0.1:8443] proxy_protocol v2     ← Caddy's PROXY-aware listener

web.sites[].upstreams:  addr 10.0.0.5:8080            Caddy → origin directly
                        pool panel                    Caddy resolves the pool's backends (via: direct)
                        pool wings1, via: wayhouse    Caddy → 127.0.0.1:<port> → listener web-up-wings1 → pool wings1

Wayhouse stays on the public ports and does not parse HTTP. The entry route (a port or sni route to the web pool with proxy_protocol: v2) delivers a PROXY v2 header with the real client address, then the client’s bytes untouched. Behind Caddy, a plain addr or a via: direct pool is reached by Caddy itself; a via: wayhouse pool goes back through a generated loopback listener, so balancing, health checks, drains and tunnel backends apply to the origin. Caddy puts the client address in X-Forwarded-For; on the via: wayhouse hop the TCP peer is Caddy’s loopback address. Fields, defaults and the port allocation are in docs/05 (“Web sites”); GET /web is in docs/06.

04 – Transport & client-IP preservation

TCP

• Accept: accept4() with SOCK_NONBLOCK; SO_REUSEPORT sharding per worker.
• Destination IP of the connection: for dst-based subdomain routing without a socket per IP, the listener is bound with IP_FREEBIND / ip_nonlocal_bind to a routed prefix; the concrete destination IP of the accepted connection comes from getsockname(). (In transparent mode getsockname() already returns the real IP the client addressed.)
• Peek for routing: recv(MSG_PEEK) up to peek_max_bytes / peek_timeout_ms. If the client sends nothing first (server-speaks-first), the default route applies immediately.
• Upstream connect: non-blocking, connect_timeout. TCP_NODELAY set; the client’s options are not “inherited” but set from config.
• Data pump: Linux splice() (socket→pipe→socket, zero-copy — ADR 17). Fallback (non-Linux, or a pipe2 failure): one 32 KiB buffer per direction, try_read/try_write. Pipe size and SO_RCVBUF/SO_SNDBUF use the kernel defaults — not exposed as config; the simpler fixed-default path covered every case actually hit so far.
• Half-close: propagate shutdown(SHUT_WR) in one direction, keep the other direction running until it too closes.
• Timeouts: connect, idle (no byte in either direction), optional max-lifetime.
• Keepalive: optionally enable TCP_KEEPALIVE toward the backend to detect dead sessions in NAT-free internal networks.

UDP

UDP has no connection — the proxy builds the “session” concept itself.

• Session key: (src_ip, src_port, dst_ip, dst_port) (the 4-tuple).
• Receive: recvmmsg() in batches (fixed at 16 datagrams/syscall, RECV_BATCH — not config-exposed, ADR 18) on SO_REUSEPORT sockets, one loop per worker.
• Send: sendmmsg() toward the backend for consecutive datagrams of one session, and toward the client in the reply pump (up to 16 per syscall, ADR 31).
• Receiving on a whole prefix (for dst routing for games with no protocol hint, see 03): the listener does not bind a socket per destination IP but one wildcard socket and enables IP_PKTINFO / IPV6_RECVPKTINFO. Per datagram the cmsg yields the actual destination address (ipi_addr / ipi6_addr); then the dst match runs against the LPM trie. Replies must carry the same source address — the destination IP on send is set via cmsg (IP_PKTINFO) or through the session socket bound to the client IP. Requirement: the prefix is routed to the proxy host (no per-address NDP/ARP needed) and net.ipv6.ip_nonlocal_bind or IP_FREEBIND permits the bind.
• One upstream socket per session, bound to the backend via connect(2):
  • Replies arrive directly on the right socket without a table lookup.
  • The kernel filters foreign senders.
  • sendmmsg() toward the backend in batches.
• Session table: HashMap<4-tuple, SessionRef> thread-local per worker (no locks). Slab allocator for Session.
• Idle timeout: timing wheel; any activity pushes the entry forward. Default 30–120 s, configurable per route (turn-based games need more).
• Caps: max sessions global / per worker / per backend; new-session rate per source IP.
• ICMP: treat “port unreachable” from the backend as a passive health signal.
• Fragmentation / MTU: no reassembly; oversized replies use normal IP fragmentation. IP_MTU_DISCOVER toward the backend as needed.

QUIC/DTLS

Treated as opaque UDP. Migration (client port change) breaks the session without QUIC-CID awareness → an optional quic sniffer that reads the connection ID and uses it as the session key instead of the 4-tuple. Later stage only.

Passing the client IP to the backend

By default the backend only sees the proxy IP. Options, selectable per pool:

1. PROXY protocol (recommended when the backend supports it)

• TCP: v2 binary header (or v1 text) before the first payload to the backend. Carries the real source/destination address. The backend (or its framework) must parse it.
• UDP: the v2 header is prepended to the first datagram of the session; subsequent datagrams have no header. Requires backend support (e.g. a library that interprets the first payload as a PROXY header). Option proxy_protocol: v2-udp.
• Security: backends must accept PROXY headers only from the proxy IP (otherwise clients can spoof IPs).

2. Transparent mode (Linux TPROXY)

• Proxy socket with IP_TRANSPARENT; the upstream connection uses IP_TRANSPARENT + binds the client IP as its source. The backend sees the real client IP with no protocol change.
• Prerequisite: routing set up so the backend’s return traffic goes back through the proxy (policy routing / ip rule + iptables -t mangle or nftables, or the backend’s default gateway = the proxy). More work in the network design, but 100% transparent.
• Required capability: CAP_NET_ADMIN (or CAP_NET_RAW), no full root.

Config

Set transparent: true on a TCP or UDP listener (Linux only). It is mutually exclusive with prefix (both derive the per-datagram destination).

TCP — wayhouse:

• binds the listen socket with IP_TRANSPARENT (so it accepts connections a TPROXY rule redirected to a non-local address; getsockname() on the accepted socket still returns the original destination, which feeds dst / port routing exactly as a normal bind does);
• for every upstream connection — pool or resolver target — opens the backend socket with IP_TRANSPARENT, bind()s the real client ip:port as its source, then connects.

UDP — wayhouse:

• binds the listen socket with IP_TRANSPARENT + IP_RECVORIGDSTADDR / IPV6_RECVORIGDSTADDR and reads the original destination ip:port from the recvmsg control message (this, not IP_PKTINFO, carries the redirected port), feeding dst / port routing and the session key;
• binds the per-session upstream socket to the real client ip:port with IP_TRANSPARENT;
• sends replies from a per-session IP_TRANSPARENT socket bound to the original destination ip:port, so the client sees them coming from the address it addressed.

If the client and backend address families differ the upstream source bind is skipped and a normal bind/connect is used (logged). v4 and v6 are both supported.

Network setup (example)

Redirect inbound game traffic to the proxy’s port 7777 and mark it, then route marked traffic locally:

# nftables: TPROXY inbound game ports to the local proxy
table inet tproxy {
  chain prerouting {
    type filter hook prerouting priority mangle; policy accept;
    ip daddr 198.51.100.0/24 tcp dport 7777 tproxy to :7777 meta mark set 1
    ip daddr 198.51.100.0/24 udp dport 7777 tproxy to :7777 meta mark set 1
  }
}

# policy routing: locally deliver anything with mark 1
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

The backend’s return traffic must come back through the proxy host — either make the proxy the backend’s default gateway, or add an ip rule on the backend network that sends the client prefixes back via the proxy. Without that the client gets replies straight from the backend IP and the connection stalls.

3. No preservation

• The backend sees the proxy IP. Sufficient when anti-cheat/logic does not need the client IP or gets it elsewhere (in the game login token). Documented default behavior.

Socket tuning (as built — kernel defaults, not config knobs)

None of these ended up as config options: every socket uses the OS default buffer/pipe sizes, and the batch size is a compile-time constant. Simpler than the originally-planned per-parameter config surface, and nothing so far has needed to override a kernel default.

SO_REUSEPORT lets any same-user socket join a port, so before binding a listener’s worker sockets the proxy probes each bind address with a socket that does not set it. A port some other process already holds is refused: startup fails, and a reload keeps the previous listener running and reports the failure (wayhouse_listener_bind_failures_total); every reload, even of an unchanged config, retries a listener whose bind failed. The probe is not atomic with the bind, so two instances starting at the same instant can both pass it.

OS limits

• Raise the FD limit (RLIMIT_NOFILE) (≥ 2 × expected connections).
• Large ephemeral port range toward the backend (net.ipv4.ip_local_port_range) — only matters in non-transparent mode; in transparent mode the socket binds the client IP.
• net.core.somaxconn, net.core.netdev_max_backlog, nf_conntrack (or disable conntrack for the proxy path via NOTRACK).

05 – Configuration

Principles

• One declarative YAML file is the source of truth. Discovery adapters and the admin API only supplement the backend lists and states at runtime.
• Validate before activating. A faulty config is rejected; the running one stays active.
• Hot reload via SIGHUP or file watch. Listeners are reconciled by name: an added listener is spawned, a removed one is stopped, and one whose definition changed (bind, protocol, routes, …) is stopped and re-spawned. A listener whose definition is unchanged keeps running untouched. backend_sources discovery refresh tasks are reconciled the same way (added / removed / re-parameterised → started / stopped / restarted).
• Environment-variable interpolation (${VAR}) for secrets/tokens.

Schema (reference)

Implemented subset (roadmap phase 2 + phase 3 routing, partial). wayhouse-config currently accepts a reduced, flatter schema: pools[].targets or a pools[].source naming a backend_sources[] entry (static / dns_srv / consul / kubernetes / tunnel, flat fields, refresh_interval_sec); balancer: round_robin | least_conn | consistent_hash | weighted (scalar, not an object) — consistent_hash also reads a pool-level hash_on: src_ip | src_ip_port (default src_ip), and weighted a pool-level weights: { "ip:port": N } map (weight >= 1, default 1), each rejected on the other balancers; health_check.type: tcp_connect | udp_probe | none with send_hex / expect_hex_prefix for udp_probe (see “Health checks” below); per_backend.max_sessions; and proxy_protocol: none | v1 | v2 | v2-udp (scalar). v1/v2 are TCP-only, v2-udp UDP-only; the form must match the transport of the listeners that statically route to the pool (validated).

A listener maps to a pool either with the pool: <name> shorthand or with a priority-ordered routes: list — [{ match, action }], first match wins, action: { pool: <name> } — but not both. Implemented matchers: match: { type: always }, { type: client_cidr, cidrs: [<prefix>, ...] } (source IP), { type: dst, cidrs: [<prefix>, ...] } (destination IP the client connected to; a plain listener sees only its bind IP, but a UDP listener with prefix: <cidr> serves a whole routed prefix on one IP_PKTINFO socket and replies from the address that was hit), { type: port, ports: [<int> | "lo-hi", ...] } (destination port from the accepting socket), { type: first_bytes, prefix: "hex:ff.." | "ascii:..", length: { min, max } } — a ≤ 512-byte prefix and/or a byte-count window on the connection’s first bytes (TCP MSG_PEEK / first UDP datagram; at least one of prefix / length), and { type: sni, host: ["exact", "*.suffix", ".suffix"] } — the server_name from the peeked (not terminated) TLS ClientHello, TCP listeners only. The { type: sniffer, sniffer: <name>, host: [...] } matcher parses (a listener may use several sniffers, see docs/03-routing.md) but matches nothing until a sniffer is loaded — no sniffers are built in; the name is checked at listener start, not by validate(). The sniffer loader is Phase 9; regex-over-first-bytes is a Phase 9 sniffer concern, not a first_bytes sub-form.

External resolver (phase 4, slices 1–4): a top-level resolvers: list of { name, type: http|grpc, endpoint, timeout_ms, on_error: reject|fallback_route|stale_ok, proxy_protocol?, target_connect_timeout_ms?, target_idle_timeout_sec?, cache? } and a route action: { resolver: <name> } (exactly one of pool / resolver per action). The proxy POSTs {listener, src, dst, sni?, first_bytes_b64, routing_key?} and expects {pool?, target?, ttl_sec?} — target (“ip:port”) wins over pool and connects straight to that instance (no pool / health / cap). proxy_protocol: none | v1 | v2 | v2-udp (default none) is the PROXY protocol header to prepend to a target connection — there is no pool to read it from — so the backend still sees the real client IP; a resolver-chosen pool uses that pool’s own proxy_protocol. Same transport rule as pools: v1/v2 need TCP listeners on that resolver’s routes, v2-udp needs UDP. target_connect_timeout_ms (default 300) / target_idle_timeout_sec (default 90) are the connect / idle timeouts for a target connection or UDP session — again, no pool to read them from; a resolver-chosen pool uses its own. fallback_route continues the route list on failure; reject drops; stale_ok serves the last (expired) cached answer if there is one, else drops. Optional cache: { key: [<part>, ...], positive_ttl_sec, negative_ttl_sec, max_entries } — a TTL’d LRU keyed by the joined parts (src_ip / src_ip_port / sni / routing_key / first_bytes:a:b); a response ttl_sec overrides positive_ttl_sec; a request missing a key part bypasses the cache. For type: grpc the endpoint is an http://host:port URI and the contract is proto/resolver.proto (wayhouse.resolver.v1.Resolver/Resolve).

Push resolver: POST /route-hint (admin API) with { "src_ip": "...", "pool": "...", "ttl_sec": 30 } records a short-lived src_ip → pool hint; a listener with route_hint: true applies it before its route list. pool must exist; ttl_sec 1..=3600.

Listener options: prefix: <cidr> (UDP only) → prefix mode as above, with a wildcard bind; freebind: true (TCP only) → bind with IP_FREEBIND / IPV6_FREEBIND; transparent: true (TCP or UDP, Linux) → TPROXY mode: IP_TRANSPARENT on the listen socket, the original destination read per connection / datagram, and the real client address bound as the upstream source (needs CAP_NET_ADMIN; mutually exclusive with prefix); route_hint: true → consult the push resolver.

Filter chain (phase 7): allow: [<cidr>, …] / deny: [<cidr>, …] on a listener (TCP or UDP) are checked against the client source IP before routing. deny is checked first and wins; a non-empty allow makes the listener default-deny for any source it does not cover. A blocked connection / new UDP session is dropped silently (no error reply — no reflection) and counted by wayhouse_filter_blocked_total{listener,filter="acl"}. Established UDP sessions are not re-checked per datagram. allow / deny are compiled to a radix trie, so large block lists (bogons + a threat feed) match in bounded time.

rate_limit: { per_ip: { rate, burst }, per_net: { rate, burst } } (at least one of per_ip / per_net) is a token bucket on new connections / new UDP sessions, checked right after the ACL. rate is permits/second sustained, burst the bucket capacity (defaults to rate). per_net aggregates by the client’s /24 (IPv4) or /64 (IPv6). A permit is taken only when every configured bucket can afford it; excess is dropped silently and counted by wayhouse_filter_blocked_total{listener,filter="rate_ip"|"rate_net"}. Bucket state is per proxy instance (size it per node behind anycast HA). Each table is capped at 200 000 buckets (one for IPs, one for networks) to bound memory; idle buckets are pruned first, but if a flood of distinct sources (an IPv6 /64 sweep, spoofed UDP) fills a table with live buckets, a source with no bucket yet is refused (fail closed) until some expire, which can also turn away legitimate new clients while the flood lasts. Those refusals are counted as rate_ip / rate_net like any other over-rate drop, and the proxy logs a rate limiter table is full warning (at most once a minute per listener and table) so the two can be told apart.

Per-source concurrent cap: per_source: { max_per_ip, max_per_net } (at least one) bounds how many connections / UDP sessions are live at once from one client IP / one /24 (v4) / /64 (v6) — where rate_limit bounds the rate of new ones. Checked after rate_limit, before allocation; over the cap ⇒ silent drop + wayhouse_filter_blocked_total{listener,filter="src_conn_ip"|"src_conn_net"}. The slot is released when the connection closes / the UDP session idles out.

GeoIP filter: geo: { allow: [CC], deny: [CC] } on a listener (ISO 3166-1 alpha-2 country codes, case-insensitive) is checked on the client source IP right after the CIDR ACL. Same precedence as allow/deny: deny wins, a non-empty allow is default-deny, and an IP the database can’t place is admitted only when there is no allow list. Requires settings.geo_db — the path to a MaxMind Country .mmdb (e.g. MaxMind’s free GeoLite2-Country); --check and startup fail if it can’t be opened, and a listener whose DB somehow isn’t loaded fails closed. Blocked ⇒ wayhouse_filter_blocked_total{listener,filter="geo"}. Startup-only, like settings.workers.

Sniffer loader (phase 9; config schema slice 2, the wasmtime loader slice 3, live dir rescanning slice 4 — all landed): settings.sniffers: { dir, call_timeout_ms, max_memory_bytes, modules: [{ name, sha256, config? }] }. Absent ⇒ no sniffers load and a sniffer: route never matches. dir is a directory of *.wasm modules, scanned at startup and rescanned on every reload (added modules load, removed ones drop, changed ones recompile — swapped in atomically, no listener restart); call_timeout_ms (default 20) bounds a sniffer’s wall-clock time per call via wasmtime epoch interruption; max_memory_bytes (default 16 MiB) caps a call’s instance memory; modules: optionally pins each module’s SHA-256 for supply-chain verification — when non-empty, a dir entry not listed there (or whose hash doesn’t match) is refused (and, on a rescan, keeps the previous sniffer set rather than applying a half-updated one). modules[].config is an optional opaque string handed to that sniffer on every sniff call (the sniffer parses it itself — e.g. a match pattern for regex-firstbytes); a module needs a modules[] entry (hence its sha256) to carry a config. validate() rejects an empty dir, a zero call_timeout_ms / max_memory_bytes, a non-64-hex-char sha256, and an empty config string. call_timeout_ms / max_memory_bytes themselves, and the settings.sniffers block appearing/disappearing, are startup-only — the wasmtime::Engine and its epoch-ticker thread are built once (like settings.workers). Modules within an already-configured dir can also be managed over HTTP: GET/POST /admin/sniffers + DELETE /admin/sniffers/{name} + POST /admin/sniffers/{name}/rollback (swaps in the kept previous version, docs/sniffers.md “Updates and rollback”) on the instance’s own admin API (409 if settings.sniffers is absent). An upload is checked before it is written and written atomically: anything that is not a loadable module (not wasm, has imports, lacks the memory / alloc / sniff exports, declares no ABI version or another one than the host’s, fails to instantiate, empty) is 400, a body over 8 MiB is 413, and on an instance with settings.sniffers.modules pins an unlisted name or a different sha256 is 409. A bad .wasm file already in dir is logged and skipped by every rescan when modules is not set. When modules pins are set, a pin violation, a pinned module that fails validation, or a pinned path that is not a readable file (a directory, a dangling symlink) fails the whole scan (a pin whose file has not been uploaded yet is fine): the process refuses to start (naming the module), and a reload keeps the previous sniffer set. Uploads are fanned out fleet-wide via wayhouse-aggregator’s POST/DELETE /fleet/sniffers[/{name}] and the admin GUI’s Sniffers page — see docs/sniffers.md “Installing by hand”.

Tier-2 regional health fabric (phase 13, docs/10 “Tier 2” — built): settings.failure_domain (a string identity — this instance’s reachability-equivalence class, e.g. an AZ or region) and settings.gossip: { bind, seeds?, quorum_fraction?, psk } must be set together, or not at all — validate() rejects either one alone. bind / seeds[] are UDP socket addresses; seeds is optional (any subset of a domain’s live members is enough to join, membership then discovers the rest) and defaults to empty. quorum_fraction (default 0.66) must be > 0.5 and <= 1.0 — the fraction of the domain that must report a backend down before that verdict can override this instance’s own “up” reading; a same-or-under-half quorum could contradict itself between two overlapping majorities, hence the > 0.5 floor. psk (a pre-shared key, must not be empty) authenticates every gossip datagram with an HMAC-SHA256 tag. When the mesh itself lands, it will only ever be able to push a backend’s flag down, never revive it — a backend returns healthy only on this instance’s own local rise streak, and a Tier-1 force-down (AdminState::Disabled) always wins over the domain view. Startup-only, like settings.workers.

Fleet grouping: settings.group (optional string) is this instance’s self-reported organization path — e.g. "eu/frankfurt/cluster-a" — pushed to wayhouse-aggregator alongside its IngestPayload so the admin GUI can render a grouped/tree fleet view. /-separated, non-empty segments, no leading/trailing /; validate() rejects anything else. Purely a display label — never consulted by routing/forwarding, and independent of failure_domain (which is about health-fabric membership, not fleet organization).

Global caps: settings.limits: { max_connections, max_udp_sessions, max_new_sessions_per_sec } are process-wide ceilings (all optional; omit for no cap). max_connections / max_udp_sessions bound the live counts across every listener; max_new_sessions_per_sec is a token bucket (burst = the rate) over new connections and new UDP sessions combined. A new connection / session that would breach a cap is dropped before it is allocated (existing ones keep running) and counted by wayhouse_filter_blocked_total{filter="max_conn"|"max_udp"|"max_new_rate"}. Startup-only, like settings.workers.

Web sites (schema 3): the optional top-level web: section describes HTTP(S) sites published through a Caddy that sits behind a Wayhouse pool (design: issue #297, spec docs/superpowers/specs/2026-10-15-web-routing-design.md). It is validated and resolved as Config.web and needs schema_version: 3. The only routing it adds are the loopback listeners for via: wayhouse upstreams (below); everything else is read by wayhouse-web-sync through GET /web (docs/06) and rendered into Caddy.

web:
  pool: web                       # pool in front of Caddy (default "web"); must set proxy_protocol: v2
  listen: { http: 127.0.0.1:8080, https: 127.0.0.1:8443 }   # where Caddy listens (defaults shown)
  trusted_proxy: []               # addresses / CIDRs Caddy trusts PROXY headers from
  internal_bind: 127.0.0.1        # loopback only (default shown)
  internal_ports: "19000-19999"   # lo-hi, at most 1000 ports (default shown)
  sites:
    - hosts: [panel.example.com, "*.apps.example.com"]
      upstreams:
        - addr: 10.0.0.5:8080     # a plain address, or
        - pool: panel             # a pool (via defaults to direct), or
        - { pool: wings1, via: wayhouse, port: 19010 }   # port pins the loopback listener
      tls: { cert: /certs/panel.pem, key: /certs/panel.key }   # served by Caddy (optional)
      upstream_tls: skip          # verify | skip | { ca_file: <path> }; absent = plain HTTP
      health_path: /healthz       # active health check path on each upstream (optional)

Rules: pool must exist and set proxy_protocol: v2 (Caddy’s PROXY listener rejects plain connections). Hosts are plain hostnames or *.suffix (labels of a-z0-9_-; no port, no IP literal, no leading-dot form), lowercased, and unique across sites; each site needs a host and an upstream. An upstream is exactly one of addr / pool; via (direct | wayhouse) and port apply only to a pool, and port only with via: wayhouse, inside internal_ports, one port per pool and one pool per port, and not one of listen’s ports; a pool used via: wayhouse must not set proxy_protocol (the next hop is plain HTTP). health_path starts with / and has no whitespace, control characters, { or }. internal_bind must be a loopback address.

listen may be a routable address: on a gateway node Caddy cannot listen on loopback only. trusted_proxy is a list of IPs or CIDRs (a bare address is a /32 or /128); with a web pool on another node, put that node’s address in it, because Caddy’s proxy_protocol allow must contain the peer. tls.cert and tls.key are PEM files that wayhouse-web-sync reads (paths on its filesystem), both non-empty; without tls Caddy has no certificate for the site’s hosts and serves them over plain HTTP. upstream_tls is per site: skip turns off verification (acceptable on an authenticated tunnel), ca_file is a path on Caddy’s side. health_path must start with /. A non-loopback listen with an empty trusted_proxy logs a warning, as does a web.pool that no TCP listener routes to (--check, startup and reload).

Listeners for via: wayhouse upstreams. For each distinct pool used with via: wayhouse the validator appends a TCP listener web-up-<pool> to the config’s listeners, bound to internal_bind:<port> with one always route to that pool. Caddy’s upstream is that loopback address, so the connection takes the normal path: pool balancing, health checks, drains, tunnel backends. Several sites sharing a pool share one listener. The port is the pin if there is one, otherwise the first 8 bytes of the SHA-256 of the pool name (big-endian) modulo the range size, moving on to the next free port when taken. Taken means: another pool’s pin or generated port, an operator TCP listener bound to the same address or to the wildcard address of the same family, or the admin address. Adding or removing another pool does not move an existing pool’s port, except when two names hash to the same port; pools are then handled in name order, so pin a port if you need it fixed whatever else changes. A pin on a port an operator listener or the admin API already holds is an error, as are a listener name that clashes with web-up-<pool> and a range with no free port left. Reload reconciles these listeners by name like any other.

Client address. Caddy sets X-Forwarded-For from the PROXY v2 header, so an origin sees the client address in HTTP with either via mode. On the via: wayhouse hop the TCP peer is Caddy’s loopback address, which is expected: Wayhouse does not parse PROXY headers inbound and none is needed for HTTP.

Plugin routes (schema 2): plugin_routes: { type: sni } (or { type: sniffer, sniffer: <name> }) on a TCP listener lets enabled controller plugins add routes to it (docs/plugins.md “Routes”). They are matched by the hostname the connection announces, appended after the listener’s own routes (so any operator route, including an always fallback, wins), and each backend gets a one-target pool named plugin-<hash of the address>. Only meaningful with --controller; the field needs schema_version: 2.

UDP first-packet gate: first_packet_gate: true on a UDP listener makes it create a session only when the first datagram is positively recognised — a sniffer hint that is not reject, or a first_bytes route on the listener that matches the datagram. Applied before the route_hint lookup (a spoofable src_ip hint must not bypass it). Unrecognised datagrams are dropped with no session and no reply, counted by wayhouse_datagrams_dropped_total{reason="first_packet_gate"}. Requires at least one first_bytes route or a sniffer (else it would drop everything).

A UDP session reads the routed pool’s idle_timeout_sec once when it is created. UDP affinity (a client that comes back after its session was evicted reaches the same backend) comes from the pool: use balancer: consistent_hash with hash_on: src_ip | src_ip_port. There is no per-listener affinity key and no sticky table (removed in #56: it was per worker, so SO_REUSEPORT kept only ~34% of clients on 4 workers). A round_robin pool gives a returning client a new backend. See config.example.yaml.

Handshake-only sniffers (UDP, #131): the quic, wireguard, openvpn, raknet and teamspeak3 sniffers recognise only a flow’s handshake datagram. The route is decided when a session opens, not per connection, so a flow whose session was evicted by idle_timeout_sec (or whose NAT mapping changed, or that reached this proxy after a restart or an ECMP move) reopens on a transport datagram that no sniffer recognises. Without a gate it falls to the next matching route; with first_packet_gate: true it is dropped until the client handshakes again (WireGuard rekeys after about two minutes). Per protocol: keep the pool’s idle_timeout_sec well above the keepalive (the config warns below 60 s, at load, reload and --check); a catch-all always route after such a sniffer that goes to a different pool is rejected unless first_packet_gate: true is set; and use route_hint, or a pool with balancer: consistent_hash (hash_on: src_ip), so a reopened session still reaches the same backend. The check goes by the sniffer’s module name.

Admin API auth (phase 10+11 slice 10): settings.admin.auth_token (a flat string, not the target schema’s auth: { mode, token } object below) gates every admin API request except GET /healthz with a bearer check. None (the default) leaves the API open — network-boundary-only auth, same as always. A single shared secret, not RBAC/mTLS — appropriate for gating a control-plane API that’s meant to stay behind its own network boundary regardless; needed once something calls in from outside that boundary, which today means wayhouse-aggregator’s intent-verb fan-out (docs/10 “The aggregator”), given the same token to present.

Admin API TLS (2026-10-02): settings.admin.tls: { cert, key } (PEM chain, leaf first; PEM private key — both required) serves the admin API over HTTPS. Startup-only like listen; the files are re-read every 30 s, so a renewed certificate needs no restart. wayhouse --check loads the pair; errors name settings.admin.tls.cert/.key. Optional keys under tls bound the TLS handshakes: max_pending (512), max_pending_per_source (16), new_per_source_per_sec (20; 0 = no rate limit), new_per_source_burst (64); see docs/12 “Native TLS”. The admin URL reported to --aggregator becomes https://<listen>, so the certificate must be valid for the listen address (an IP SAN), and the aggregator needs --ca-file for a private CA. With --controller every instance gets the same YAML, so the paths are the same on every host while each host’s file must hold a certificate for its own listen address. See docs/12 “TLS for the fleet services”.

The full schema below is the target.

# Optional: the minimum config schema this document needs (whole number, 1 or
# higher; absent = 1). See "Schema version" below.
schema_version: 1

# global
settings:
  workers: 0                 # 0 = number of CPU cores
  shutdown_grace_sec: 30     # wait this long for in-flight conns on SIGTERM
  admin:
    listen: "127.0.0.1:9900"
    auth: { mode: "bearer", token: "${ADMIN_TOKEN}" }   # or mode: mtls
  metrics: { path: "/metrics" }
  log:
    format: "json"
    connection_log: { enabled: true, sample: 1.0 }
  limits:
    max_connections: 500000
    max_udp_sessions: 1000000
    max_new_sessions_per_sec: 50000   # new conns + UDP sessions/s (burst = rate)

# reusable filters
filters:
  - name: block-bogons
    type: deny_cidr
    cidrs: ["10.0.0.0/8", "192.168.0.0/16", "100.64.0.0/10"]
  - name: conn-rate
    type: rate_limit
    scope: src_ip            # src_ip | src_/24 | global
    new_per_sec: 50
    burst: 100

# backend sources (phase 8) — a pool takes its backends from `targets:` OR a
# named `source:`, never both. Level-triggered: the adapter returns the current
# address set; the runtime diffs it against the live set. A refresh that errors
# or returns empty keeps the last-known-good set. `static` is folded into the
# pool's targets at load time; the dynamic kinds each get one control-plane
# refresh task (never on the data path).
backend_sources:
  - name: static-eu
    type: static
    targets: ["10.1.0.11:7777", "10.1.0.12:7777"]
  - name: k8s-match
    type: kubernetes            # LIST .../endpointslices for the service, plus a watch for instant updates
    service: "match-server"
    namespace: "games"          # default: "default"
    port_name: "game"           # optional; else each slice's first port
    api: "https://kubernetes.default.svc"   # default; SA token + CA read in-pod
    refresh_interval_sec: 10   # k8s: resync interval; changes arrive via the watch
  - name: consul-eu
    type: consul               # GET /v1/health/service/<service>?passing=true, plus a blocking query for instant updates
    service: "match-server"
    consul_addr: "http://127.0.0.1:8500"    # default
    tag: "prod"                # optional
    consul_token_file: "/run/secrets/consul-token"   # optional ACL token (X-Consul-Token), re-read as it rotates
    refresh_interval_sec: 10   # consul: resync interval; changes arrive via the blocking query
  - name: srv-us
    type: dns_srv              # resolves the SRV record; port from the record
    record: "_game._udp.us.internal.example.com"
    refresh_interval_sec: 10
  - name: home-origin
    type: tunnel               # phase 14 (built) — the `tunnel` BackendSource.
                                # Resolves backend addresses a `wayhouse-agent`-managed
                                # origin behind a WireGuard tunnel has registered
                                # with the controller's backend-peers registry.
    pubkey: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="  # origin's WG pubkey
    refresh_interval_sec: 10

# pools
pools:
  - name: match-eu
    source: static-eu
    balancer: { strategy: consistent_hash, hash_on: src_ip }
    affinity: { table_ttl_sec: 120 }
    health_check:
      type: udp_probe          # tcp_connect | udp_probe | none | http
      send_hex: "01"
      expect_hex_prefix: "02"
      interval_sec: 2
      timeout_ms: 500
      rise: 2
      fall: 3
    per_backend:
      max_sessions: 200
      max_new_per_sec: 20
    proxy_protocol: "none"     # none | v1 | v2 | v2-udp
    connect_timeout_ms: 300
    idle_timeout_sec: 90

  - name: match-us
    source: srv-us
    balancer: { strategy: least_conn }
    health_check: { type: tcp_connect, interval_sec: 2, timeout_ms: 400, rise: 2, fall: 3 }

  - name: source-query
    source: static-eu
    balancer: { strategy: round_robin }
    idle_timeout_sec: 10

# resolvers (external routing logic)
resolvers:
  - name: matchmaker
    type: grpc                 # grpc | http
    endpoint: "https://matchmaker.internal:8443"
    timeout_ms: 40
    proxy_protocol: "none"     # none | v1 | v2 | v2-udp — header for a `target` result
    target_connect_timeout_ms: 300  # connect / idle timeout for a `target` result
    target_idle_timeout_sec: 90     #   (no pool to read them from)
    cache:
      key: ["first_bytes:0:16"]     # e.g. a session-token prefix
      positive_ttl_sec: 30
      negative_ttl_sec: 2
      max_entries: 200000
    on_error: "reject"         # reject | fallback | stale_ok

# listeners + routes
listeners:
  - name: public-udp
    bind: "0.0.0.0:7777"
    protocol: udp
    filters: ["block-bogons", "conn-rate"]
    peek_max_bytes: 64
    routes:
      - match: { type: always }
        action: { resolver: matchmaker }
      - match: { type: always }        # fallback if resolver on_error=fallback
        action: { pool: match-eu }

  - name: public-tls
    bind: "0.0.0.0:443"
    protocol: tcp
    peek_max_bytes: 2048
    peek_timeout_ms: 200
    routes:
      - match: { type: sni, suffix: ".eu.example.com" }
        action: { pool: match-eu }
      - match: { type: sni, suffix: ".us.example.com" }
        action: { pool: match-us }
      - match: { type: always }
        action: { pool: match-eu }

  - name: public-mc
    bind: "0.0.0.0:25565"
    protocol: tcp
    peek_max_bytes: 512
    routes:
      - match: { type: sniffer, name: minecraft, host: "survival.example.net" }
        action: { pool: match-eu }
      - match: { type: always }
        action: { pool: match-us }

  - name: source-27015
    bind: "0.0.0.0:27015"
    protocol: udp
    routes:
      - match: { type: first_bytes, prefix: "hex:FFFFFFFF" }
        action: { pool: source-query }
      - match: { type: always }
        action: { pool: match-eu }

  # Raw UDP with no protocol hint: subdomain by destination IP (see docs/03 scheme A)
  - name: raw-udp
    bind: "[::]:7777"                     # wildcard bind, one socket
    protocol: udp
    prefix: "2001:db8:ace:1::/64"         # enables IP_PKTINFO / IPV6_RECVPKTINFO;
                                           # routes by the real per-datagram destination
    routes:
      - match: { type: dst, cidrs: ["2001:db8:ace:1::1/128"] }   # survival.example.net
        action: { pool: match-eu }
      - match: { type: dst, cidrs: ["2001:db8:ace:1::2/128"] }   # creative.example.net
        action: { pool: match-us }
      # No `always`/catch-all route and no `reject` action — there isn't one;
      # an unmatched destination IP simply has no matching route and is
      # dropped (no_route).

  # IPv4-only variant: subdomain by port (scheme B), SRV hands out the port.
  # One listener, one `bind: "host:lo-hi"` port range (F1.4) — one real socket
  # per port, all sharing this listener's routes; the `port` route matcher
  # picks the pool by the port the datagram actually arrived on. Ports in the
  # range with no matching route (anything but 30001/30002 here) are dropped.
  - name: raw-udp-v4
    bind: "0.0.0.0:30000-30099"
    protocol: udp
    routes:
      - match: { type: port, ports: [30001] }
        action: { pool: match-eu }
      - match: { type: port, ports: [30002] }
        action: { pool: match-us }

A kubernetes source reads the service’s EndpointSlices (discovery.k8s.io/v1, selected by the kubernetes.io/service-name label) and merges them: ready endpoints only (an unset ready counts as ready), duplicates across slices collapsed, FQDN slices skipped, and a slice without the wanted port ignored. Its service account needs list and watch on endpointslices in the namespace; deploy/k8s/45-wayhouse-rbac.yaml is a ready-made Role. Without watch the source logs one warning per outage and keeps converging on the poll interval, so a missing grant slows updates but never breaks discovery. The watch resumes from the version of the last list and reopens itself when the API server ends it; a 410 Gone triggers an immediate list, which renews the version. A failing watch retries every 5 s, doubling up to 60 s. The service-account token is read from its file again every 60 s, so the kubelet’s rotation of the bound token (about hourly) is followed without a restart.

A consul source lists the passing instances of the service and then holds a blocking query (?index=<X-Consul-Index>&wait=300s) on the index of its last list; a new index triggers a list at once, and the poll interval stays as the resync net. An index that goes backwards, or a response without one, is treated as a reset and gets a fresh list. A failing query retries like the Kubernetes watch (5 s doubling to 60 s). With ACLs enabled, put the token in a file and point consul_token_file at it; the file (not the token) is what the YAML and GET /config carry, and it is read again every 60 s. The token needs service:read on the service and node:read on the nodes running it. The service name and the tag are URL-escaped.

Both pushed signals are coalesced: after the first change the source is fetched once it has been quiet for 500 ms (at most 5 s after the first change), so a rolling update of many pods costs one fetch, not one per event.

Validation rules (excerpt)

• A pool needs exactly one of targets: / source:; source must point to a backend_sources[].name. refresh_interval_sec >= 1. dns_srv needs record; consul / kubernetes need service (consul_token_file, if given, must not be empty); a Kubernetes api (which receives the pod’s service-account token) and a consul_addr used together with consul_token_file must be https:// or http:// to a loopback address, with no user info; tunnel needs pubkey (a base64-encoded 32-byte WireGuard key).
• Every action.pool / action.resolver must exist.
• Every listener needs at least one route. There is currently no warning for a route list with no trailing always — an unmatched connection/datagram is silently dropped (no_route); a “did you forget a catch-all route?” lint is a plausible small future addition, not built today.
• proxy_protocol: v2-udp only together with protocol: udp (for a pool, and for a resolver whose routes are on UDP listeners); v1/v2 only with TCP.
• A resolver’s timeout_ms, target_connect_timeout_ms and target_idle_timeout_sec must all be > 0.
• transparent: true (TCP or UDP) may not be combined with prefix, and may not be set on a listener whose pool: or a route’s pool action uses a tunnel backend source (the client and tunnel address families can differ). A resolver route can still return such a pool at runtime; there the proxy connects without the client’s source address.
• Every entry in a listener’s allow / deny must be a valid CIDR.
• rate_limit, if present, needs at least one of per_ip / per_net, each with rate >= 1.
• per_source, if present, needs at least one of max_per_ip / max_per_net, each >= 1.
• Every settings.limits.* value, if present, must be >= 1 (0 would block all traffic — omit the key for no cap).
• first_packet_gate: true is UDP-only and needs at least one first_bytes route or a sniffer on the listener.
• A UDP listener with a handshake-only sniffer (quic, wireguard, openvpn, raknet, teamspeak3) must not have a later always route to a different action unless first_packet_gate: true is set (#131).
• A listener geo filter requires settings.geo_db, a non-empty allow or deny, and 2-letter country codes; the DB file must open at startup / --check.
• consistent_hash requires hash_on.
• weights is accepted only with balancer: weighted; every key must parse as ip:port and every value must be >= 1.
• settings.sniffers.dir must not be empty; call_timeout_ms / max_memory_bytes must be >= 1; each modules[].sha256 must be a 64-char hex digest and name must not be empty; a modules[].config, if given, must be non-empty.
• settings.failure_domain and settings.gossip must both be set, or neither; gossip.bind / each gossip.seeds[] entry must be a valid socket address; quorum_fraction must be > 0.5 and <= 1.0; psk must not be empty.
• match.type: dst needs the destination address to be known: on TCP it comes from getsockname() for free; on UDP it needs either transparent: true (any dest) or prefix: <cidr> (dest within that prefix) on the listener — there is no separate recv_dst_addr flag, prefix/transparent themselves turn on the recv path. freebind is TCP-only and is rejected together with prefix (a UDP prefix listener needs the routed prefix reachable to the host, not IP_FREEBIND).
• match.type: port matches the destination port of the accepting socket; its own ports: list supports a "lo-hi" range.
• A listener’s bind is either one host:port socket address, or a host:lo-hi port range (F1.4) — one real socket per port (TCP: one per worker per port; UDP: same, SO_REUSEPORT-shared), all sharing that listener’s routes/filters/pool selection; a route’s port matcher still sees the real port a connection/datagram arrived on. Capped at 1024 ports per range (a typo like 0-65535 would otherwise try to open tens of thousands of sockets). Mutually exclusive with prefix (which needs exactly one wildcard socket).
• Bind addresses must not overlap between listeners (same IP:port:proto, checked per port of a range too); a prefix bind must not cover a single bind address of another listener.
• Numeric ranges: timeouts > 0, rise/fall ≥ 1, TTLs ≥ 0.

Schema version

schema_version (top level, optional) is the minimum config schema the document needs; absent means 1. A build supports up to the schema in wayhouse_config::version::CONFIG_SCHEMA_VERSION (currently 3). A document that asks for more is refused with config schema_version N is newer than this build supports (max M), on startup, on a file reload (the running config stays active) and on a controller-delivered revision. A string, 0 or a fractional value is an error that says so. deny_unknown_fields stays on, so an older node also rejects a newer field loudly instead of ignoring it.

Bump rule for contributors: a PR that adds a config field, even an optional one, raises CONFIG_SCHEMA_VERSION and adds the field’s dotted path to FIELD_SINCE (wayhouse-config/src/version.rs) with the new number. The parser then refuses a document that uses that field without a schema_version that covers it, so nobody adopts a new field by accident while older proxies still run.

Mixed fleets. Proxies that register with the controller (the tunnel source) report the newest schema they understand (max_config_schema) and how often they re-register. POST /config (and POST /config/promote/{revision}, for a staged revision) answers 422 when the document’s schema_version is above the lowest value reported by any live proxy: one that re-registered within three of its own intervals, so a stopped or deleted proxy stops counting. A live proxy that reports nothing predates the field and cannot parse a document that carries the schema_version key, so it holds such documents back until it upgrades or expires; a document without the key still goes through. If the controller cannot read its proxy registry the check fails closed (503). Upgrade every proxy before submitting a document that declares schema_version or uses a field introduced by the new version. Proxies that do not register (no tunnel) are not tracked: they still refuse a document they cannot parse and keep their previous config.

Reload semantics

Health checks

• health_check.type defaults to tcp_connect, which can never succeed against a UDP-only backend. A pool that only UDP listeners route to and that leaves type out is therefore rejected at load: pick udp_probe (with send_hex, and expect_hex_prefix if the reply is recognisable) or none. A pool shared with a TCP listener keeps the default; an explicit type is always accepted.
• none runs no active probe, for a game with no probe payload. Health then comes from passive observations alone: a UDP ICMP port-unreachable, a failed upstream send or a TCP connect failure counts against a backend, and a backend marked unhealthy is given another chance after one interval_sec (re-admitted after rise such intervals), since nothing else could ever bring it back. A truly dead backend therefore flaps: down on passive failures, back up after rise intervals, down again on the next failure.
• For UDP, a successful send() is not a passive success (it succeeds whether or not anything listens). The first reply datagram of a session is.
• A proxy-side shortage (EMFILE, ENFILE, ENOMEM, ENOBUFS, EADDRNOTAVAIL) while opening a socket, for a probe or for a session, is not counted for or against the backend.
• Probes run in the background, at most 256 at once; the first check of each backend is spread over one interval_sec. A slow probe does not delay the others, and a backend is never probed twice at once.
• idle_timeout_sec must be greater than 0 (as for the other timeouts).

06 – Operations & observability

Metrics (Prometheus)

Listener / connections

• wayhouse_listener_connections_total{listener,result} – result = accepted|no_route|sniffer_reject (sniffer_reject: a sniffer sniffer returned a reject hint for the first bytes; the connection is dropped before routing). ACL / rate-limit / geo / cap rejections do not get a result value here — they increment wayhouse_filter_blocked_total instead (see “Security / filter chain” below), and a resolver miss/error is folded into no_route rather than its own value. No protocol label — split by listener name instead (a listener has one fixed protocol).
• wayhouse_active_connections{listener} (gauge) — no protocol label.
• wayhouse_active_udp_sessions{listener} (gauge)
• wayhouse_connection_duration_seconds{listener} (histogram) — no pool label (a connection’s pool can change across a reload; the listener is stable).
• wayhouse_session_setup_seconds{listener,phase} — planned, not implemented; see “Planned / not yet built” below.

Throughput

• wayhouse_bytes_total{listener,dir} – dir = c2s|s2c. No pool label.
• wayhouse_packets_total{listener,dir} (UDP). No pool label.
• wayhouse_datagrams_dropped_total{listener,reason} — v0 reason = no_route|no_backend|upstream_bind|upstream_send|outside_prefix|draining|reply_bind|first_packet_gate|sniffer_reject|pending_full (outside_prefix: prefix-mode listener, datagram destination not in prefix; first_packet_gate: first_packet_gate listener, first datagram not recognised; sniffer_reject: a sniffer sniffer returned a reject hint — no session, no reply; pending_full: a new session’s external resolver call is still in flight and the per-worker pending cap (1024 sessions, 1 MiB buffered) or the per-session buffer (4 datagrams) is full)

Upstream / pool

• wayhouse_pool_backends{pool,state} (gauge; state = healthy|unhealthy|draining|disabled)
• wayhouse_backend_active_sessions{pool,backend} (gauge)
• wayhouse_backend_connect_errors_total{pool,backend,kind} – kind = timeout|refused|unreachable
• wayhouse_healthcheck_total{pool,backend,result}
• wayhouse_lb_selections_total{pool,strategy,result} – strategy = round_robin|least_conn|consistent_hash|weighted; result = ok|no_backend|at_capacity
• wayhouse_route_hints_applied_total{listener} – a POST /route-hint entry decided routing for a connection / UDP session
• wayhouse_proxy_protocol_headers_total{pool,version} – version = v1|v2|v2-udp; a PROXY protocol header was prepended to an upstream connection (TCP) or the first datagram of a session (UDP v2-udp). pool="(resolver target)" for a pool-less resolver target connection (the header form comes from the resolver’s proxy_protocol:)

Backend discovery (phase 8)

• wayhouse_discovery_refresh_total{pool,kind,result} – kind = dns_srv|consul|kubernetes|tunnel; result = ok|empty|error|withdrawn. One per refresh attempt. empty / error keep the last-known-good backend set (the pool is never cleared by a failed refresh). withdrawn (tunnel only) means the controller deleted or expired an origin this proxy had seen: the pool’s set is cleared.
• wayhouse_discovery_backends{pool} (gauge) – addresses returned by the pool’s source at its last successful refresh.

Security / filter chain

• wayhouse_accept_errors_total{listener} (counter, #174) – a TCP accept failed, typically EMFILE (out of file descriptors). The worker pauses 50 ms, doubling to 1 s, until an accept succeeds, and logs at most one accept failed warning per second with a suppressed count; this counter carries the full rate.
• wayhouse_filter_blocked_total{listener,filter} – filter = acl | geo | rate_ip | rate_net | src_conn_ip | src_conn_net | max_conn | max_udp | max_new_rate; a connection / new UDP session was dropped by the pre-routing filter chain — acl = source-IP allow/deny, geo = GeoIP country allow/deny (or the DB failed to load and the listener fails closed), rate_ip / rate_net = the per-listener per-IP / per-/24 (per-/64) token bucket was empty, src_conn_ip / src_conn_net = the per-listener concurrent per-source connection / session cap was reached, max_conn / max_udp / max_new_rate = a process-wide settings.limits cap was hit

Sniffers (phase 9)

• wayhouse_sniffer_calls_total{name,result} – result = ok | unrecognised | timeout | trap | bad_output; one increment per sniff() call through the WASM loader. timeout = the epoch-interruption deadline (settings.sniffers.call_timeout_ms) fired; trap = any other WASM trap or an instantiation failure; bad_output = the sniffer returned a result the host couldn’t decode as a RouteHint.
• wayhouse_sniffer_call_seconds{name} (histogram) – wall-clock time of one sniff() call, instantiation included.

Resolver

• wayhouse_resolver_requests_total{resolver,result} – ok|empty|timeout|error
• wayhouse_resolver_cache_total{resolver,result} – hit|hit_negative|miss|stale|uncacheable

Proxy internals

• wayhouse_config_reload_total{result} / wayhouse_config_version (gauge, timestamp)
• wayhouse_listener_bind_failures_total{listener} – a reload could not (re)bind the listener (port taken by another process, no permission, bad address). The previous listener, if any, keeps running; every later reload retries (even of an unchanged file). Startup refuses to run instead.
• wayhouse_build_info{component,version,commit} (gauge, always 1) — set once at startup; the same label set on wayhouse (component="wayhouse") and every fleet binary. commit is a 12-char git SHA baked in at build time (crates/wayhouse-http/build.rs): the WAYHOUSE_GIT_SHA environment variable if set (the Docker build arg, since the build has no .git), else git rev-parse, else "unknown" (e.g. a source tarball).
• wayhouse_fd_open (gauge, no labels) — this process’s open file descriptor count (/proc/self/fd on Linux; absent elsewhere), sampled every 5 s by a small background task (crates/wayhouse/src/procinfo.rs), independent of the health-check sweep.
• wayhouse_fd_limit (gauge, no labels) — this process’s RLIMIT_NOFILE soft limit (getrlimit, via nix), sampled once at startup (it doesn’t change at runtime).
• wayhouse_tls_handshakes_refused_total{reason="per_source"|"rate"} (counter) — TLS connections to the admin API closed at the door: the source already had its cap of handshakes in flight, or was opening connections faster than its rate. Counted in wayhouse-http (its name lives in wayhouse_http::tls, not metrics_defs.rs, which wayhouse-http cannot depend on). Exposed on wayhouse‘s /metrics and on the fleet binaries’ (below). Present only when that listener serves TLS.
• wayhouse_tls_handshakes_evicted_total (counter, no labels) — pending handshakes dropped to make room at the global cap (same notes).
• wayhouse_gossip_members (gauge, no labels) — current SWIM member count in this instance’s Tier-2 gossip mesh (phase 13, docs/10 “Tier 2”, wayhouse-core:: gossip). Present only when settings.gossip is set.
• wayhouse_gossip_messages_total{direction="sent"|"received"} (counter, phase 13) — gossip datagrams that passed HMAC verification.
• wayhouse_gossip_auth_rejected_total (counter, no labels, phase 13) — gossip datagrams dropped for a missing/invalid HMAC tag; never trusted, never forwarded to the SWIM state machine.
• wayhouse_gossip_stale_rejected_total (counter, no labels) — authentic gossip datagrams dropped because their sender timestamp is more than 30 s from this node’s clock (a replay, or an instance with a skewed clock; keep NTP running).
• wayhouse_gossip_version_rejected_total (counter, no labels, #185) — authentic, fresh gossip datagrams dropped because their magic or version byte is not this build’s gossip version: a peer on an incompatible release. Dropped before decoding; the first one per process is logged as a warning.
• wayhouse_gossip_registers_dropped_total{reason="future"|"non_member"} (counter) — authentic gossip health registers dropped because their timestamp is more than 60 s ahead of this node’s clock (the origin’s clock runs fast; keep NTP running) or their origin is not a current member. Each origin also gets a warn log line at most once a minute.
• wayhouse_protocol_mismatch_total{route_group="controller"|"aggregator"|"raft"|"proxy"} (counter, #185) — component requests refused with 426 because the caller’s X-Wayhouse-Protocol major differs, is not <major>.<minor>, or (on component routes) is missing. Emitted by the controller, aggregator and proxy; see 10 “Versioning”.
• wayhouse_backend_domain_down{pool,backend} (gauge, 0/1, phase 13) — whether the Tier-2 domain quorum is currently overriding this backend to down. Independent of, and unable to clear, the backend’s own local healthy flag (wayhouse_pool_backends’s healthy/unhealthy counts already reflect the combined result).

Added RTT: no built-in RTT SLO metric exists yet — make bench’s added p50/p99 (vs. NFR N1/N2) is the closest thing today, measured out-of-band, not exported as a /metrics series. See “Planned / not yet built” below.

wayhouse-web-sync (web proxy, web-proxy.md)

Served on the sync’s own GET /metrics (--metrics-listen, default 127.0.0.1:9904).

• wayhouse_web_sync_last_ok (gauge) — unix time of the last cycle that left Caddy on the intended configuration (loaded, or already identical). Alert on time() - this.
• wayhouse_web_sync_loads_total (counter) — documents Caddy accepted.
• wayhouse_web_sync_rejects_total (counter) — documents Caddy refused (4xx); it keeps its previous configuration, and the sync does not resend the same document.
• wayhouse_web_sync_errors_total{stage="read"|"render"|"send"} (counter) — cycles that failed before Caddy had a new document: read Wayhouse (/web, /pools), render (including unreadable certificate files), send (Caddy unreachable, timeout, 5xx).
• wayhouse_web_sync_last_render_seconds (gauge) — how long the last render took.

Controller ACME (--acme), on the current leader’s /metrics (a follower emits none of these), not on wayhouse’s (these are registered by wayhouse-controller, not in metrics_defs.rs): wayhouse_acme_cert_expiry_seconds{site}, wayhouse_acme_issue_total{result} and wayhouse_acme_held; see acme.md.

Planned / not yet built

Documented here as real future work, not implemented — none of these exist in metrics_defs.rs today, so don’t expect them on /metrics yet:

• wayhouse_resolver_latency_seconds{resolver}, wayhouse_session_setup_seconds{listener,phase}, wayhouse_worker_busy_ratio{worker} — plausible finer-grained latency / saturation instrumentation, deferred until a real debugging need shows the existing aggregate metrics (wayhouse_resolver_requests_total, wayhouse-bench’s added-latency numbers) aren’t enough to explain a slowdown. Not worth the hot-path cost speculatively.
• wayhouse_resolver_cache{resolver,state} / wayhouse_buffer_pool_exhausted_total — dropped from the plan: the former was a duplicate of wayhouse_resolver_cache_total above (typo’d as a second metric), the latter described a buffer-pool architecture that was never built (per-worker fixed buffers replaced it early on — see docs/09 “Key libraries”).

Connection logs (structured, JSON)

One event on session close (plus optionally on start), sampleable:

{
  "ts": "2026-09-03T10:48:12Z",
  "event": "session_close",
  "listener": "public-udp",
  "protocol": "udp",
  "src": "203.0.113.7:51343",
  "route": "route#1",
  "resolver": "matchmaker",
  "pool": "match-eu",
  "backend": "10.1.0.12:7777",
  "bytes_c2s": 184320,
  "bytes_s2c": 942080,
  "pkts_c2s": 1440,
  "pkts_s2c": 1460,
  "duration_ms": 812345,
  "setup_ms": 6,
  "close_reason": "idle_timeout",
  "proxy_protocol": "none"
}

close_reason: client_close|backend_close|idle_timeout|drain|limit|backend_down|error.

Tracing (optional, OpenTelemetry)

Trace only the setup phase (not the byte stream): span session.setup with child spans peek, route.match, resolver.call, backend.connect. The trace ID can be carried in the PROXY v2 TLV if the backend should correlate it.

The proxy’s own health endpoints

• GET /healthz – the process is alive.
• GET /readyz – config loaded & valid, at least one listener bound, and not drained. Returns draining (503) after POST /admin/drain.
• GET /metrics – Prometheus.
• GET /config – plaintext view of the active snapshot (listeners + pools + draining / active_conns). Implemented (phase 5). GET /pools – per-pool backend list with health / state= / active count.
• GET /web – the resolved web: section as JSON for wayhouse-web-sync: pool, listen.{http,https}, trusted_proxy, internal_bind, and per site its hosts and upstreams in final form ({addr}, {pool, via: "direct"}, or {pool, via: "wayhouse", local: "127.0.0.1:<port>"} for the generated loopback listener). via: direct pools are resolved by the reader from GET /pools. Same bearer token and TLS as the other routes; 404 when the config has no web: section. Follows reloads. Per site it also carries the optional tls (cert, key), upstream_tls ("verify", "skip" or {"ca_file"}) and health_path; read by wayhouse-web-sync.
• GET /sessions[?listener=&pool=&proto=tcp|udp&src=<ip>] – plaintext list of every live proxied connection / UDP session: id, transport, listener, client (peer=) and local (local=) address, chosen pool= / backend=, and age=. Backed by a small id → metadata registry on the connection tracker (one brief lock per connection at open / close, nothing on the per-byte path). Point-in-time; the optional query params filter by exact listener / pool, transport, or source IP. Implemented.
• POST /admin/drain / POST /admin/undrain – take this instance out of / back into LB rotation by flipping readyz, without stopping the data path. Implemented (phase 5).
• POST /route-hint {src_ip, pool, ttl_sec} – push-resolver hint (scheme C); applied by listeners with route_hint: true. Implemented.
• PATCH /pools/{pool}/backends/{addr} {state: enabled|draining|disabled} – set an operator backend state. Implemented (phase 5). draining / disabled remove the backend from new-session selection (including UDP affinity) immediately; existing sessions keep running. Carried across a config reload by address. GET /pools shows the current state= per backend.

Backend health checks

• Active: tcp_connect / udp_probe (send/expect bytes). Parameters: interval, timeout, rise, fall. Checks run in the control plane, the result → a new snapshot. An http kind (ping a sidecar health port) is planned, not implemented — HealthCheckKind only has TcpConnect/UdpProbe today; deferred until a real backend that exposes an HTTP health endpoint needs it.
• Passive: the data path reports connect refused/timeout, early RST, ICMP unreachable. After n errors within t seconds → backend unhealthy (faster than the active check).
• Recovery: back to healthy only after rise successful active checks.

Graceful draining & deployments

Flow for swapping a backend without dropping players:

1. Register the new backend (POST /pools/{p}/backends) → healthy after rise checks.
2. Set the old backend PATCH state=draining → the LB assigns no new sessions there; a client whose resolver answer pointed at it is re-resolved once the cached answer expires.
3. Wait until wayhouse_backend_active_sessions = 0 or drain_deadline is reached.
4. After the deadline: end remaining sessions with close_reason=drain.
5. Remove the old backend.

For a proxy instance restart:

1. Set readyz to “false” (POST /admin/drain; POST /admin/undrain reverts) → the upstream LB/anycast takes the instance out of rotation. The data path keeps running; watch active_conns (in GET /config) fall.
2. Grace period: no new accepts, existing sessions keep running.
3. After settings.shutdown_grace_sec (default 30) close remaining sessions, exit the process.
4. SIGINT / SIGTERM triggers exactly this flow (implemented, phase 5): the TCP accept loops and the UDP recv loops stop taking new work immediately; in-flight TCP connections and established UDP sessions keep running and are waited on (UDP sessions until they idle out); once all have finished — or the grace period expires — the process exits. New datagrams to a draining UDP listener are dropped (wayhouse_datagrams_dropped_total{reason="draining"}). A config reload that replaces a UDP listener behaves differently: the old workers close their sockets at once and their sessions end, so the kernel hashes every flow to the replacement group, which opens a fresh session (under the new config) on the client’s next datagram. Nothing is dropped as draining, and a chatty client cannot hold a drain open. A listener that is removed, or whose bind address changed, still drains as described above.

Multi-instance operations (HA)

Production is many identical, independent wayhouse instances across hosts / AZs / regions, fronted by anycast or an L4 load balancer. No shared data-plane state (ADR 4): lose an instance and the LB / anycast spreads its clients onto the rest; affected sessions reconnect.

• Config distribution: either the file-based approach below, or the phase 10+11 controller (see “Fleet control plane” below) — bake into the image (change = rolling redeploy), render the file from a store (ConfigMap / consul-template / Ansible) and let file-watch pick it up in seconds, or point every instance at a wayhouse-controller with --controller <url>. Either way the structural config is hot-reloaded; validate-before-swap keeps a bad config from taking an instance down (the controller rejects it at submission time instead, one hop earlier — see below).
• Backend membership comes from phase-8 discovery (backend_sources: DNS SRV / Consul / k8s Endpoints), not per-instance file edits. Every instance runs the same source config and converges independently; a source outage freezes the set at last-known-good rather than draining a pool fleet-wide.
• Operator intent (drain / add / remove a backend, route hints) is still applied per instance, either directly against that instance’s admin API or fanned out through wayhouse-aggregator (see below), and is not persisted or replayed after a restart — moving intent into the controller’s revision log is phase 12, not this release.
• transparent: true needs the backend’s return path through the same instance that owns the connection (per-flow-consistent ECMP / anycast). PROXY protocol has no such constraint — prefer it when the fronting layer can rehash.
• Aggregate /metrics and /pools yourself in Prometheus for per-instance detail; for a single fleet-wide read/operate surface, see wayhouse-aggregator and wayhouse-ui below — there is still no fleet-wide metrics aggregation.

Fleet control plane (phase 10+11)

Three additional, independent binaries. Originally a single-tier PoC (phase 10+11); the docs/10 standalone/slave hierarchy, intra-tier HA, adoption, staged/canary rollout, RBAC, and moving operator intent into the controller’s revision log are phase 12 — all built, documented per-flag in the subsections below. All three serve unauthenticated GET /healthz for liveness regardless of their auth settings below.

GET /metrics on the fleet binaries. Each serves a Prometheus endpoint (wayhouse_http::metrics) with wayhouse_build_info{component,version,commit} and the TLS handshake counters above. It is gated like the rest of the binary’s API: on wayhouse-controller and wayhouse-aggregator by --auth-token (Authorization: Bearer, open when no token is set), unless a dedicated --metrics-token (at least 16 bytes) is given: then /metrics accepts only that token (not the admin token) and the token opens nothing else, so Prometheus never holds the admin secret. wayhouse-ui has only --metrics-token, because a scraper cannot hold the UI’s browser session. A wayhouse-ui with a login configured and no --metrics-token does not serve /metrics at all; an open UI (no login) serves it open. Per-request or per-store metrics for these binaries are not built; the per-instance proxy surface stays on each wayhouse.

wayhouse-controller — structural config distribution

One --auth-token-gated (optional) surface, backed by an embedded sled store that never forgets a revision (ADR 20):

• POST /config — submit a raw YAML config (the same document a wayhouse --config file would hold). Validated with the same wayhouse_config::parse_str a file reload runs; a rejected submission (422, JSON error body) leaves the current revision untouched — the revision only advances on success. Response body is {"revision": N} (the new revision only appears here, not as a header).
• GET /config — the current revision’s raw text, with X-Config-Revision header. 404 before any submission.
• GET /config/subscribe?since=<revision> (SSE) — catch-up range then a live tail; wayhouse --controller <url> consumes this to hot-reload. A lagging subscriber just re-runs the catch-up query against the store — no delivery state kept on the writer side.
• GET /config/revisions — history (revision, size, whether it’s current).
• GET /config/revisions/{revision} — a past revision’s raw text.
• GET /config/revisions/{revision}/diff[?against=<revision>] — a line diff against current or another revision.
• POST /config/rollback/{revision} — re-submits that revision’s exact bytes through the same validate-then-store path POST /config uses; rollback never rewrites history, it creates a new revision with old content.

A wayhouse instance opts in with --controller <url> (+ --controller-token if the controller requires one) instead of --config <file> — the two are mutually exclusive. On a controller outage wayhouse keeps running its last-applied config and retries the subscribe connection with capped exponential backoff (500ms → 30s); nothing about the data path pauses.

Role (phase 12 slice 1): --role standalone (default) is everything above unchanged. --role slave --parent-url <url> [--parent-token <token>] makes this controller a relay: it never accepts a write of its own (POST /config and POST /config/rollback/* both 403, "this controller is a slave tier...") and instead seeds from the parent’s GET /config at startup, then holds a GET /config/subscribe connection to the parent and re-lands every revision it receives in its own store (its own local revision numbers — not required to match the parent’s, only the config text is shared). A wayhouse instance, or a further slave tier, points at this controller exactly as it would at a standalone one — the relay is transparent below this tier. Losing the parent freezes this tier on last-known-good with the same reconnect-with-backoff behavior wayhouse’s own --controller client has; it never promotes itself to standalone.

Operator intent log (phase 12 slice 3): alongside /config*, every wayhouse-controller also serves POST /intent and GET /intent/subscribe, a second sled-backed revision log (its own <data_dir>/intent database) covering the pool-scoped phase-5 admin verbs — backend add ({"op":"backend_add","pool":...,"addr":...}), backend remove (backend_remove), backend admin state (backend_patch, state one of enabled/draining/disabled), and route-hint (route_hint, {"src_ip":...,"pool":...,"ttl_sec":...}, default ttl_sec 30). POST /intent validates the op’s shape (addr/IP parse, known state) before accepting it — there’s no wayhouse_config::validate() equivalent for a bare op, so this is that check, one hop earlier than broadcasting it. A wayhouse instance running with --controller automatically also subscribes to /intent/subscribe and applies every op it receives through the same RuntimeHandle calls the admin API’s own handlers use — an intent op is a new source for an existing mutation, never a new code path. Not part of this log: whole-instance drain/undrain (targets one instance, not “every instance with pool X” — use the aggregator’s targeted fan-out or the instance’s own /admin/drain) and resolver pins (still open, docs/01). The slave-role write gate applies to /intent too, and (phase 12 slice 4) a slave tier also relays the intent log from its parent, the same way it relays config — a slave controller now holds a full copy of both of its parent’s logs while still never originating either one itself.

Intra-tier HA (phase 12 slice 6): --ha-node-id <id> + --ha-peers id=host:port,... (identical on every replica; an entry may instead be id=https://host[:port], see docs/12 “HA replicas over TLS”) turn a standalone tier into an N-node Raft group (embedded openraft) replicating both the config and intent logs together. POST /config and POST /intent propose a Raft entry and only return once it’s committed; a replica that isn’t the current leader transparently forwards the request to whichever one is (never a redirect — every existing client stays unaware HA exists). Reads (GET /config//config/subscribe//intent/subscribe/etc.) are served by whichever replica gets the request, straight from its own local copy — not a linearizable read, a deliberate relaxation justified in docs/10 “Intra-tier HA (design)”. --ha-token gates the new peer-only /raft/* routes (append, vote, snapshot) — a separate secret from --auth-token. HA combines with --role slave (2026-10-05): only the Raft leader subscribes to the parent and proposes each relayed revision into the group; the relay cursor (the highest parent revision absorbed, one per log) is replicated with the log, so a newly elected leader resumes where the group left off and a duplicate proposal is skipped. Start every replica with the same --role slave --parent-url .... POST /admin/adopt is refused on an HA controller (the role is a per-node setting).

Staged / canary rollout (phase 12 slice 7): POST /config?stage=canary &group=<name> submits a revision visible only to a subscriber reporting that group; a plain POST /config (no stage) is unaffected — immediately visible to everyone, exactly as before this feature existed. GET /config?group=<name> and GET /config/subscribe?since=<revision>&group= <name> show what’s visible to that group: the highest revision that’s either promoted or staged to it. POST /config/promote/{revision} makes an existing revision visible to everyone (404 if it doesn’t exist) — this is the one operation that changes an existing revision rather than creating a new one. GET /config/revisions gained promoted/canary_groups fields per entry. One group per submission in this release. Not available for /intent (config only).

Adoption (phase 12 slice 5): POST /admin/adopt {"parent_url":"...","parent_token":"..."} flips a running standalone tier to slave without a restart. Refused with 409 unless this tier’s config and intent stores are both still empty (a tier with existing history must be replaced by a fresh one to join a hierarchy, per docs/10 “Adoption”) or it’s already a slave. On success, seeds from the new parent’s current config and starts relaying both logs, exactly like a --role slave boot — the response body’s seeded_config_revision is the parent’s revision number seeded from (null if the parent’s config log was empty at that moment). Same --auth-token gate as /config*//intent*.

wayhouse-aggregator — fleet reads and operational fan-out

One --auth-token-gated (optional) surface fed by every instance’s own push, plus a --instance-token this aggregator presents going back out to each instance’s admin API (a separate secret from --auth-token — one gates calls in, the other authenticates calls out):

Hierarchy (phase 12 slice 2): --parent-url (+ --tier-name, required; --parent-token; --parent-push-interval-sec, default 10) makes this tier also push its own merged view — every instance it currently knows, individually, with its name rewritten "{tier_name}/{instance}" — up to a parent aggregator’s POST /ingest, on the same interval shape wayhouse’s own --aggregator push uses. A parent needs no configuration to accept this: a child aggregator’s push looks exactly like a proxy’s own push, just under a namespaced instance name, so it shows up in the parent’s /fleet/* views with no special-casing. Namespacing only affects what’s pushed upward — this tier’s own local /fleet/* view and admin_url-based fan-out still use the un-namespaced names.

• POST /ingest — an instance’s periodic self-reported summary (pool/backend health + admin state, session counts, its own admin_url). Latest-write-wins, never persisted — every fact here is a proxy’s own state, re-pushed on the next tick, so restarting the aggregator loses nothing durable.
• GET /fleet/pools / /fleet/sessions — every known instance’s ingested summary, each entry carrying last_seen_ms_ago.
• GET /fleet/healthz — per-instance staleness (stale past 30s, ~3x wayhouse’s default 10s push interval) and the product version each instance runs (empty from a build that predates the field). The UI backend checks a registry sniffer’s min_proxy against these versions on install and update.
• GET /fleet/subscribe (SSE) — the same summaries, pushed on change (debounced 150ms so a burst of near-simultaneous instance pushes collapses into one resend, not one per push).
• POST /fleet/instances/{instance}/drain / /undrain — targeted fan-out to that instance’s own POST /admin/drain//admin/undrain; 404 if the instance isn’t known.
• POST /fleet/pools/{pool}/backends — broadcast to every known instance’s own POST /pools/{pool}/backends.
• PATCH / DELETE /fleet/pools/{pool}/backends/{addr} — broadcast to every instance’s own PATCH/DELETE /pools/{pool}/backends/{addr}.
• POST /fleet/route-hint — broadcast to every instance’s own POST /route-hint.

Every broadcast response is {"results": [{"instance", "status", "error", "detail"?}, ...]} — one entry per known instance, status: null (not a failed request) for one that couldn’t be reached; an instance that refuses (non-2xx) also carries detail, the first 200 bytes of its reply; a broadcast never fails or blocks on one bad instance. A wayhouse instance opts in with --aggregator <url> (+ --aggregator-token, --aggregator-instance, --aggregator-interval-sec, default 10s, and --aggregator-admin-url, the base URL the aggregator should fan out to when http(s)://<settings.admin.listen> is not reachable from it, e.g. in a container, behind NAT or a TLS terminator) — independent of --controller, pushing state and pulling config are unrelated axes.

wayhouse-ui — the operator dashboard’s BFF

A dedicated process holding both the controller’s and the aggregator’s own bearer tokens on the operator’s behalf; the browser only ever holds a session cookie (POST /ui/login//ui/logout, GET /ui/session), never a bearer token. Everything else is a thin, header-preserving proxy: /api/fleet/* → the wayhouse-aggregator routes above (--aggregator-url/ --aggregator-token), /api/config* → the wayhouse-controller routes above (--controller-url/--controller-token), and GET /ws/fleet — a browser WebSocket fed by one shared subscription to the aggregator’s /fleet/subscribe (one aggregator connection total, fanned out to every connected browser, not one per tab). An open socket does not outlive its session: it re-checks the session every 30 s and before each outgoing message, and closes with code 1008 after logout, idle timeout, max age or eviction (the check doesn’t count as activity, so a tab left open still idles out). --static-dir (default crates/wayhouse-ui/web/dist, built by make ui) serves the React/Vite/TS frontend as a fallback under whatever the API routes above don’t claim — it’s the one process, one port an operator’s browser ever talks to. Either proxy target is optional; fleet reads/config actions 503 cleanly if the corresponding --*-url was never given.

RBAC and audit (phase 12 slice 8, docs/10 “RBAC and audit (design)”): two login modes, mutually exclusive.

• --ui-password <secret> (legacy, unchanged from phase 10+11) — one shared secret, POST /ui/login {"password":"..."}, every session implicitly admin.
• --users-file <path> — multi-operator accounts, a YAML list of {username, password_hash, role} (role one of viewer/operator/ admin); login is POST /ui/login {"username":"...","password":"..."}. password_hash is an argon2 PHC string — wayhouse-ui --hash-password reads a password from stdin and prints one, the intended way to populate an entry (never put a plaintext password in the file).

With neither flag, the UI is fully open and every session is implicitly admin — same posture every other optional-auth surface in this fleet has. By default open mode is loopback-only; --insecure-no-auth permits a non-loopback --listen with no login (and logs a warning). The UI guards the operator’s own browser against other sites: a state-changing request or WebSocket upgrade is refused (403) when Sec-Fetch-Site is cross-site or same-site, or, when the browser sends no Sec-Fetch-Site, when Origin is null or names another host; in open mode every request’s Host must also be a loopback name or address (against DNS rebinding; this Host check applies only when the UI listens on loopback). Requests without those headers (curl, scripts) pass. Names the UI forwards to the aggregator or controller (backend, pool, sniffer) are percent-encoded to stay one path segment, and . / .. are refused.

Sessions expire: --session-idle-timeout-secs (default 1800) and --session-max-age-secs (default 43200, also the cookie’s Max-Age), with at most --max-sessions (default 1000) held at once. POST /ui/login is rate-limited per client address and per username (429 + Retry-After).

Three roles gate three route groups: viewer (every GET — fleet reads, config/revision reads/diffs, GET /ws/fleet), operator (+ the phase-5 intent verbs — drain/undrain, backend add/patch/delete, route-hint), admin (+ config submit/rollback/promote). A session below a route’s minimum role gets 403 (distinct from 401, which means no valid session at all).

Audit: every write wayhouse-ui proxies carries an X-Actor: <username> header (--ui-password mode never sets it — there’s no per-session identity there). wayhouse-controller records it per revision — GET /config/revisions’ new actor field is the durable half of this audit trail. wayhouse-aggregator logs (instance/pool, actor, verb) via tracing for the fan-out verbs and forwards the header on to each instance — a convenience, not a durable record (this aggregator holds no durable state by design).

Fronting layer: anycast vs. L4 load balancer

Rules of thumb: prefer PROXY protocol over transparent whenever the fronting layer may rehash a live flow (anycast reconvergence, LB scaling) — a rehashed TCP flow that lands on a new instance is a reconnect either way, but PROXY protocol keeps client-IP preservation working without a symmetric return path. Keep UDP idle timeouts short (tens of seconds) so a moved client re-establishes quickly.

Capacity planning per instance

Size an instance by the scarcest of:

• New-session rate. Each new TCP connection / UDP session does one Pool::acquire, one upstream connect(2), one task spawn (see the latency ledger in HANDOVER.md). Load-test with wayhouse-bench --connections and tcpkali; set settings.limits.max_new_sessions_per_sec to ~70 % of the measured knee and alert before it.
• Concurrent sessions / FDs. ~1 FD per client + 1 per upstream, plus the per-session buffers (2×64 KB UDP, 2×32 KB TCP). Set max_connections / max_udp_sessions under the file-descriptor ulimit with headroom; watch wayhouse_fd_open against wayhouse_fd_limit.
• Bandwidth. Single-stream throughput is near line rate (the pump is a buffered copy); the limit is NIC / softirq. Spread interrupts (RSS) and run workers = cores.
• Headroom for failover. With N instances behind the fronting layer, plan so any single instance loss leaves the rest below ~75 % on every axis above.

Dashboards & alerts

Per-instance panels: wayhouse_active_connections / wayhouse_active_udp_sessions, wayhouse_listener_connections_total rate by result, wayhouse_bytes_total rate, wayhouse_connection_duration_seconds p50/p99, wayhouse_fd_open vs. wayhouse_fd_limit. (No worker-level busy-ratio panel yet — that metric is planned, not implemented; see above.) Fleet roll-ups: sum by (pool) (wayhouse_pool_backends{state="healthy"}), sum(rate(wayhouse_filter_blocked_total[5m])) by (filter), sum(rate(wayhouse_discovery_refresh_total{result!="ok"}[15m])) by (pool,kind).

Alerts (in addition to the list below):

• wayhouse_discovery_refresh_total{result!="ok"} continuously for > 3 refresh intervals on a pool → the source is down and the pool is frozen at last-known-good.
• wayhouse_discovery_backends drops > 50 % between scrapes → a bad discovery result; cross-check against wayhouse_pool_backends{state="healthy"}.
• count(up{job="wayhouse"}) < N_expected → an instance is gone; confirm the fronting layer took its VIP / member out.
• wayhouse_config_version stale (not advancing) across a rollout → file-watch or SIGHUP not reaching that instance.

The v2 distributed control plane (10-distributed-control-plane.md) removes the “fan it out yourself” / “re-apply after restart” caveats and adds a single fleet view + web UI.

Capacity planning / alerts

• Latency budget (NFR N1/N2): make bench (crates/wayhouse-bench, latency mode) measures the proxy’s added p50/p99 request→response latency on a single host and reports PASS/MISS vs. < 0.5 ms / < 2 ms. Run it before/after a change to catch per-connection overhead regressions.
• Concurrency ramp (partial N1/N2-under-load, informational for N4/N5): cargo run --release -p wayhouse-bench -- --mode concurrency spawns the real wayhouse binary as a separate process and ramps a real held-open connection count through it (--steps), reporting the proxy child’s own RSS/fd count and added-latency percentiles at each step — see crates/wayhouse-bench/README.md. Reaches tens of thousands of connections on one host (capped by the client’s ephemeral-port range), not the full 500k/1M N4/N5 targets, and says nothing about N3 (loopback bandwidth exceeds the 20 Gbit/s target, so it can’t be validated locally) or N9 (real HA). Aggregate throughput (N3), the full 500k conns / 1M sessions (N4/N5) and HA (N9) still need dedicated hardware, multiple hosts, and a real load generator (tcpkali, wrk2, iperf3).
• Alert: wayhouse_pool_backends{state="healthy"} < N_min per pool.
• Alert: wayhouse_datagrams_dropped_total rate > 0 (buffers too small / overload).
• Alert: wayhouse_resolver_requests_total{result!="ok"} share > 1%.
• Alert: wayhouse_fd_open / wayhouse_fd_limit > 0.8.
• Once built (see “Planned / not yet built” above): worker-busy-ratio and session-setup-latency alerts.

07 – Security & DDoS mitigation

Threat model

Filter chain (before routing, cheapest first)

1. CIDR allow/deny – static lists + an optional dynamic block list (feed/file, hot reload). O(1) via an LPM trie. An IPv4-mapped IPv6 address (::ffff:a.b.c.d) is matched as the IPv4 address it carries, so it cannot slip past an IPv4 deny list.
2. Geo filter (optional) – MaxMind Country DB (settings.geo_db), per-listener geo: { allow, deny } by ISO country code; deny wins, a non-empty allow is default-deny, an unplaceable IP is admitted only with no allow list. Fails closed if the DB is not loaded.
3. Connection/datagram rate limit – token bucket per src_ip and per /24 (against distributed single IPs). Scope configurable. Excess → drop + metric, no error reply (no reflect).
4. Global caps – max_connections, max_udp_sessions, max_new_sessions_per_sec. When reached: reject new ones, protect existing ones.

All steps run before buffer/session allocation.

TCP hardening

• Kernel: net.ipv4.tcp_syncookies=1, a sensible somaxconn, a short tcp_synack_retries.
• accept backpressure: when a worker is overloaded, throttle the accept rate instead of letting the queue explode.
• Peek timeout (peek_timeout_ms): anyone who sends nothing after connecting is dropped before a backend connect happens.
• Min-progress watchdog: a connection with no byte progress beyond idle_timeout → close.
• Optional per-source concurrent cap: per_source: { max_per_ip, max_per_net } (implemented — a live counter per client IP / /24 / /64, refused before allocation).

UDP hardening

• No blind replies. The proxy sends toward the client only after the backend has replied within an established session.
• First-packet gate (optional, per route): the first datagram must pass a first_bytes match (a known handshake prefix / a sniffer saying “valid”), otherwise no session is created. Keeps generic spoof floods off the session table.
• Session cap per src_ip / /24 (per_source, implemented — a hard concurrent counter, new sessions refused when full) and global (settings.limits.max_udp_sessions). LRU eviction of the oldest idle sessions under pressure is not done — the idle sweep reaps and new sessions are refused until room frees.
• Keep the idle timeout short where the game allows it (query pools: a few seconds).
• Receive buffers large enough that legitimate bursts do not compete with flood drops.
• Disable conntrack for the proxy path via NOTRACK (otherwise it is its own DoS surface).

Network & deployment hardening

• Backends: their own security zone; the ingress firewall allows only proxy source IPs to the game server ports. Backend egress minimal.
• The proxy runs without root: only CAP_NET_BIND_SERVICE (ports < 1024) and — if transparent mode — CAP_NET_ADMIN. Read-only rootfs, seccomp profile.
• Secrets (admin token, mTLS keys) via env/file with 0600, not in the YAML.
• Bind the admin API and metrics to a separate internal interface; never on the public listener IPs.
• Rate-limit and ACL state is per instance; behind anycast HA, size it per node (an attack is spread across all nodes).

Abuse as an amplifier – checklist

Guarded by crates/wayhouse-core/tests/amplification.rs:

• No reply to datagrams without an established session (no_unsolicited_or_duplicated_replies).
• No error / unsolicited reply from the app path to a dropped datagram — routing, ACL, rate-limit, first-packet-gate drops are all silent (dropped_datagrams_get_no_error_reply). ICMP is the kernel’s; the proxy emits none itself.
• Reply size never larger than the backend payload — the proxy forwards backend bytes verbatim and prepends nothing toward the client (reply_is_exactly_the_backend_payload).
• The rate limit applies before any state change — rejected datagrams create no session and are never forwarded (rate_limit_is_enforced_before_any_state_change).

Sniffer sandbox guarantees (phase 9)

A loaded sniffer (settings.sniffers, crates/wayhouse/src/sniffer_loader.rs) runs under wasmtime as a core WASM module with no WASI and no host imports — nothing is wired in beyond the alloc/sniff exports the sniffer itself provides. Concretely, a sniffer cannot:

• read or write the filesystem — no wasi_snapshot_preview1, no ambient authority of any kind is linked in;
• open a socket or make any network call — same reason;
• read the wall clock, environment variables, or process/thread state — nothing exposes them;
• see any byte the proxy hasn’t explicitly copied into its linear memory (the first-bytes prefix the route already peeked, capped at peek_len() ≤ PEEK_MAX) — it never sees later application bytes, and never sees another connection’s data (a fresh Store + Instance, and therefore fresh linear memory, per call);
• send anything toward the client — a Sniffer only returns a RouteHint (host / key / reject); it has no reply path, so the amplifier checklist above is untouched by sniffers entirely.

Two independent bounds cap what a misbehaving (buggy or malicious) sniffer can cost the process, checked every call, not just at load time:

• Time: wasmtime epoch interruption. A dedicated ticker thread bumps the engine’s epoch every call_timeout_ms / 2; each call sets a two-tick deadline before running, so a call still executing at the second tick traps (wayhouse_sniffer_calls_total{result="timeout"}). This is a ceiling, not an exact limit: the first tick lands anywhere after the call starts, so a call gets between call_timeout_ms / 2 and call_timeout_ms. A call well under half the timeout never traps (sniffer_loader::tests::short_calls_never_time_out_under_the_shipped_ticker).
• Memory: a StoreLimits cap (max_memory_bytes) on the Store — a sniffer that tries to grow its linear memory past the cap gets a failed memory.grow (WASM-spec -1), not more memory.

Supply chain: settings.sniffers.modules: [{ name, sha256 }] optionally pins each module’s SHA-256; when set, a dir entry that isn’t listed (or whose hash doesn’t match) fails the whole load/rescan rather than loading unpinned code.

Latency vs. NFR N1 (< 0.5 ms added; docs/01-requirements.md) — the “latency is a gate, not an assumption” decision from docs/08 Phase 9, measured end to end (real alloc/memory.write/sniff/decode round trip, a fresh Store + Instance per call, release build) against the three official sniffers (released from wayhouse-proxy/sniffers):

(Loopback numbers on the same box as crates/wayhouse-bench’s N1/N2 measurements; 2000 calls/sniffer after warmup. Reproduce with make sniffers-fetch then cargo test -p wayhouse --release wasm_boundary -- --ignored --nocapture in crates/wayhouse/src/sniffer_loader.rs.) All three pass N1 with wide margin — a fresh per-call Store/Instance (the “no state survives between connections” design from slice 3) turned out cheap enough on this hardware that the InstancePre / pooling-allocator / warm-instance-per-worker fallbacks noted in docs/08 Phase 9 were not needed. The occasional higher-than-typical max samples (hundreds of µs, still far under the 500 µs p50 gate) are consistent with ordinary scheduling jitter, not a systematic cost — no sample in any run has been anywhere near N1.

Security logging

• Events: ACL block, rate-limit trip (aggregated, not per packet), session cap reached, health flap, config reload (who/what via the admin API), resolver error bursts.
• No plaintext of game payload in logs (privacy); metadata only.

08 – Roadmap

Incremental. Each phase is usable on its own.

Status legend: ✅ done · 🔜 next · ⬜ planned.

Phase 0 – Skeleton ✅

• ✅ Project setup (Cargo workspace: wayhouse-config, wayhouse-core, wayhouse), CI (fmt + clippy -D warnings + tests), test harness.
• ✅ Config loading + validation + immutable snapshot behind ArcSwap.
• ✅ Structured logging (tracing), /healthz, /readyz, /metrics, build_info.

Phase 1 – L4 TCP proxy ✅

• ✅ TCP listener with SO_REUSEPORT, one accept task per core.
• ✅ Static listener→pool mapping, round_robin + least_conn.
• ✅ Bidirectional pump; shared idle timeout (neither direction moved bytes), teardown on error, connect timeout, half-close. Linux splice(2) zero-copy fast path landed in the perf pass (ADR 17) — socket → pipe → socket in the kernel, buffered try_read/try_write fallback elsewhere, both behind the same proxy::pump fn.
• ✅ Active tcp_connect health checks (rise/fall), passive connect-failure feedback, backend healthy/unhealthy state, per-backend session caps.
• ✅ Hot reload on SIGHUP / config-file change: rebuild snapshot, atomic swap, backend health carried over by address. Listener bind/protocol/pool-mapping changes still need a restart (full listener reconfiguration is phase 5).
• ✅ Core metrics: connections, bytes, duration, backend connect errors, pool backend counts, healthcheck results, LB selections, config reload/version.
• Result: usable as a plain TCP game server front proxy.

Phase 2 – L4 UDP proxy ✅

• ✅ UDP listener with SO_REUSEPORT, one recv task per core, worker-local (lock-free) session table. Ingress batches — one recvmmsg(2) pulls up to RECV_BATCH datagrams per wakeup (perf pass, ADR 18). Idle expiry is a single-level timing wheel (perf pass, ADR 19). Egress batches with sendmmsg(2) on both legs (perf pass, ADR 31).
• ✅ Per-session connect(2) upstream socket + a reply-pump task per session.
• ✅ Session affinity: a consistent_hash pool (hash_on: src_ip | src_ip_port). The per-worker sticky table that first provided it was removed (#56): with SO_REUSEPORT it kept only ~34% of clients on 4 workers.
• ✅ udp_probe health check (send_hex / expect_hex_prefix).
• ✅ Per-backend session caps (shared with TCP) + per-pool idle timeout; amplification guard (no reply without an established session).
• Result: covers the majority of real-time game servers.

Phase 3 – Routing intelligence ✅

• ✅ Route rule list with priorities (listeners[].routes, first match wins; bare pool: normalised to one always route).
• ✅ Matchers: always, port (destination port), client-cidr (source IP).
• ✅ first-bytes matcher — prefix (hex: / ascii:, ≤ 512 B) and/or length: { min, max }; TCP peek / first UDP datagram. (Regex-over-first-bytes moved to Phase 9 — it is a sniffer concern, not a first-bytes sub-form.)
• ✅ consistent_hash balancer (rendezvous/HRW hash, pool hash_on: src_ip | src_ip_port) — session affinity without a sticky table.
• ✅ SNI peek matcher (sni, host exact / *.suffix / .suffix; ClientHello peeked, not terminated; TCP only).
• ✅ dst matcher (destination IP vs cidrs). ✅ UDP prefix listener (prefix: <cidr>, one wildcard IP_PKTINFO socket serving the whole routed prefix, replies from the hit address) + TCP freebind. ✅ push resolver (POST /route-hint, per-listener route_hint: true).
• ✅ Sniffer API seam — wayhouse_core::sniff::Sniffer → RouteHint, the sniffer matcher, and the listener wiring that feeds a hint into routing. No built-in sniffers ship (game-specific parsing does not belong in the proxy binary); a sniffer: route only matches once a sniffer is loaded. The loader is Phase 9.
• Result: multiple games/regions behind one port (via dst / sni / first-bytes; game-protocol sniffing once sniffers land).
• ✅ F1.4’s port-range listener (docs/01-requirements.md) — bind: "host:lo-hi" (e.g. "0.0.0.0:30000-30999") spawns one real socket per port (× workers, SO_REUSEPORT-shared) under one listener config, for dedicated-server fleets that dynamically pick a port per match/instance (Agones-style hostPort ranges, Source/UE servers auto-incrementing past a taken default port) and for scheme B (subdomain-per-port) at real scale. This requirement was written down at project start and not carried into an implementation slice for a long time — caught by a documentation audit, not by design — then closed out right after. wayhouse_config::ListenerConfig::bind (primary/lowest port) + extra_binds (the rest); capped at 1024 ports per range. Mutually exclusive with prefix (which needs exactly one wildcard socket). The port route matcher (which already supported a "lo-hi" range within one listener’s routes) is what actually splits pools by port within the range — the bind range just makes the sockets exist.

Phase 4 – External routing logic ✅

• ✅ Slice 1: resolvers: config + action: { resolver: <name> }; the Resolver trait + async routing loop in wayhouse-core; HttpResolver (reqwest) in wayhouse; pool results; on_error: reject | fallback_route.
• ✅ Slice 2: result cache — configurable key (src_ip / src_ip_port / sni / routing_key / first_bytes:a:b), positive/negative TTL, max_entries LRU (lru crate), on_error: stale_ok.
• ✅ Slice 3: gRPC transport (tonic + prost, proto/resolver.proto + build.rs; CI installs protoc).
• ✅ Slice 4: Resolution.target (fixed instance, pool-less connect path in proxy.rs / listener_udp.rs — no health / cap / guard). sticky_key was dropped from the protocol (#54): nothing could recover the key on a later request, and consistent_hash plus the result cache cover affinity.
• ✅ Post-phase (data-plane completion): resolvers: reloads live — wayhouse_core::Resolvers gained ArcSwap interior mutability (like Sniffers), and the reload task rebuilds + swaps the clients when (only when) ResolverConfig differs. Trade-off: a rebuild resets each CachedResolver’s LRU cache.
• ✅ Post-phase (data-plane completion): backend_sources: reloads live — wayhouse_core::SourceManager (the discovery analogue of ListenerManager) owns one refresh task per pool source behind a private stop channel and reconciles them on every reload by diffing Snapshot::sources (pool → SourceConfig, newly echoed). Added / removed / re-parameterised entries start / stop / restart their task; a removed pool also drops its cached discovered set. wayhouse-core stays HTTP-free — the wayhouse binary supplies a SourceFactory (DiscoveryFactory) that rebuilds the concrete adapter.
• Result: matchmaker integration, token→instance routing.

Phase 5 – Operations & zero-downtime ✅

• ✅ Hot reload (SIGHUP + file watch), atomic snapshot swap. (phase 1)
• ✅ Slice 1: enabled / draining / disabled backend states — AdminState on Backend, excluded from new-session selection (incl. UDP affinity) while existing sessions drain; carried across reload by address; PATCH /pools/{p}/backends/{addr}; wayhouse_pool_backends{state=draining|disabled}.
• ✅ Slice 3: POST /admin/drain / POST /admin/undrain (flip readyz without stopping the data path) + GET /config (plaintext snapshot view with draining / active_conns).
• ✅ Slice 4: backends CRUD — POST /pools/{p}/backends {addr} / DELETE /pools/{p}/backends/{addr}; edits held in a runtime BackendOverlay layered on the file config (survives a file reload). An edit calls request_reload() and the existing reload path rebuilds via Snapshot::build_with_overlay.
• ✅ Slice 2: graceful draining of the proxy instance — a ConnTracker counts in-flight TCP connections + UDP sessions; SIGINT/SIGTERM stops the accept/recv loops and Runtime::shutdown_with_grace waits for the tracked work to finish, bounded by settings.shutdown_grace_sec (default 30). UDP listeners keep pumping established sessions until they idle out; new datagrams while draining are dropped (reason="draining").
• ✅ Slice 5: runtime listener add / remove / rebind — ListenerManager owns one task group per listener; a reload reconciles them by name (SO_REUSEPORT makes a same-bind rebind gapless).
• ✅ GET /sessions — live connection / session registry on the connection tracker (id → {proto, listener, peer, local, pool, backend, age}), exposed as a filterable plaintext admin endpoint. (data-plane completion, list C)
• ✅ Passive health signals from the data path. (phase 1)
• Result: a production-ready deploy/update cycle.

Phase 6 – Client-IP preservation ✅

• ✅ Slice 1: PROXY protocol v1/v2 (TCP) — per-pool proxy_protocol: none | v1 | v2; one header prepended to the upstream connection before any client bytes (wayhouse_core::proxy_protocol), wayhouse_proxy_protocol_headers_total.
• ✅ Slice 2: v2-UDP variant — proxy_protocol: v2-udp (UDP-only, validated) prepends the v2 binary header to the first datagram of each session; later datagrams are untouched.
• ✅ Slice 3: transparent mode (TPROXY) for TCP — transparent: true (Linux) binds the listen socket with IP_TRANSPARENT and every upstream connection with the real client ip:port as its IP_TRANSPARENT source; docs/04 carries the nftables + policy-routing setup.
• ✅ Slice 4: UDP transparent mode — IP_TRANSPARENT + IP_RECVORIGDSTADDR on the listen socket (original ip:port per datagram), a client-bound IP_TRANSPARENT upstream socket, and a per-session IP_TRANSPARENT reply socket bound to the original destination. IPV6_TRANSPARENT listen binds via the socket2 0.5 → 0.6 bump. Mutually exclusive with prefix.
• Result: backends see the real client IP (PROXY protocol or fully transparent, TCP + UDP).

Phase 7 – Security & hardening ✅

• ✅ Slice 1: CIDR allow/deny filter chain — per-listener allow / deny CIDR lists, checked on the client source IP before routing (TCP accept + UDP first datagram). deny wins; a non-empty allow is default-deny. Blocked = silent drop + wayhouse_filter_blocked_total{listener,filter="acl"}. Linear scan of the (small) Cidr list; established UDP sessions are not re-checked per datagram. (Slice 6 replaced the scan with a radix trie.)
• ✅ Slice 2: per-listener token-bucket rate limit on new connections / new UDP sessions — rate_limit: { per_ip, per_net } (rate permits/s + burst), per_net keyed by /24 (v4) / /64 (v6), checked after the ACL. A permit needs every configured bucket; excess dropped silently + wayhouse_filter_blocked_total{filter="rate_ip"|"rate_net"}. One Mutex<HashMap> per listener shared across workers, lazy prune of idle buckets. Established UDP sessions keep a scan-free steady path.
• ✅ Slice 3: global caps — settings.limits.{max_connections, max_udp_sessions, max_new_sessions_per_sec} (all optional; startup-only). Live AtomicUsize counters for TCP conns / UDP sessions + a token bucket for the new-session rate, held in one wayhouse_core::limits::GlobalLimits shared by every worker. A new connection / session over a cap is refused before allocation (existing untouched) + wayhouse_filter_blocked_total{filter="max_conn"|"max_udp"| "max_new_rate"}. RAII LimitGuard releases the slot on connection / session end.
• ✅ Slice 4: UDP first-packet gate — first_packet_gate: true on a UDP listener opens a session only when the first datagram is positively recognised (a non-reject sniffer hint, or a matching first_bytes route). Checked before the route_hint lookup; unrecognised ⇒ no session, no reply, wayhouse_datagrams_dropped_total{reason="first_packet_gate"}. validate() rejects it on TCP or with nothing to gate on.
• ✅ Slice 5: automated amplifier-checklist tests (crates/wayhouse-core/tests/amplification.rs) — no unsolicited / duplicated replies, no error reply to a dropped datagram, reply size == backend payload (proxy adds nothing toward the client), rate limit enforced before any state change. Checklist in docs/07 now ticked.
• ✅ Slice 6: ACL longest-prefix-match trie — wayhouse_config::CidrSet, a binary radix trie over address bits (v4 / v6 separate), built once per listener spawn from the allow / deny lists. Acl::permits now does a bounded bit-walk instead of a linear Cidr scan, so large threat-feed block lists cost the same as a handful of entries. Config, semantics and GET /config output unchanged.
• ✅ Slice 7: optional GeoIP country filter — settings.geo_db (a MaxMind Country .mmdb, opened once at startup / --check) + per-listener geo: { allow, deny } (ISO 3166-1 alpha-2), checked right after the CIDR ACL on the client source IP, same precedence. wayhouse_config::GeoAcl holds the decision; wayhouse_core::geo::GeoDb (dep: maxminddb) does the lookup. Fails closed if the DB isn’t loaded. wayhouse_filter_blocked_total{filter="geo"}.
• ✅ Slice 8: cargo-fuzz harnesses for the untrusted-input parsers (crates/wayhouse-config/fuzz/): extract_sni (TLS ClientHello reader), route_match (matchers + extract_sni + host patterns over a fuzzed first-bytes buffer), parse_config (parse_str on arbitrary input). Seeds in fuzz/seeds/, make fuzz, and a nightly CI job. No crashes in the initial runs (10M+ execs on extract_sni).
• ✅ Slice 9: per-source concurrent connection / session cap — per_source: { max_per_ip, max_per_net } on a listener bounds how many connections / UDP sessions are live at once from one client IP / /24 (v4) / /64 (v6), where rate_limit bounds the rate. wayhouse_core::src_conns::SourceLimiter (live counters + RAII SourceGuard), checked after rate_limit, released on connection close / session eviction. wayhouse_filter_blocked_total{filter="src_conn_ip"|"src_conn_net"}.
• ✅ Slice 10: crates/wayhouse-bench — a single-host latency/load harness (make bench) that measures the proxy’s added request→response latency (p50 / p99) against NFR N1 (< 0.5 ms) / N2 (< 2 ms), with an optional busy-connection load knob, informational single-stream throughput and a loose idle-RSS-per-connection number. N3 (aggregate throughput), N4/N5 (500k / 1M) and N9 (HA) still need dedicated hardware + a real load generator.
• Result: hardened against common L4/7 abuse; per-connection overhead measurable via make bench.

Phase 8 – Discovery & scaling ✅

• ✅ BackendSource seam in wayhouse-core (discovery.rs: trait + Discovery last-known-good cache + refresh_loop), concrete adapters in the wayhouse binary (discovery.rs: DnsSrvSource via hickory-resolver, ConsulSource / KubernetesSource via reqwest). Same HTTP-free seam as resolvers.
• ✅ Level-triggered: a source returns the current address set; the runtime diffs it against the live set. One control-plane refresh task per source (refresh_interval_sec), never on the data path (zero data-path cost — latency ledger).
• ✅ Fed through the existing snapshot rebuild (Snapshot::build_with_sources ← reload::apply): one snapshot writer. Precedence: discovered set (or file seed) ∪ overlay-added − overlay-removed, then health / admin state.
• ✅ Degrade to last-known-good: an errored / empty refresh keeps the previous set (never clears the pool) + wayhouse_discovery_refresh_total{result} / wayhouse_discovery_backends.
• ✅ Config: top-level backend_sources: list referenced by pools[].source (exactly one of targets / source). static is folded into the pool’s targets at load time.
• ✅ k8s lists the service’s EndpointSlices (discovery.k8s.io/v1) with the in-pod SA token + CA, and (post-phase-8) watches them from the listed resourceVersion: a pod change triggers a fetch at once, while the poll interval stays as the resync safety net.
• ✅ HA operations chapter in docs/06 — anycast vs. L4 LB, capacity planning per instance, dashboards & alerts.
• Result: dynamic backend fleets, horizontal scaling.
• ✅ backend_sources: live reload (SourceManager) — landed in the data-plane-completion pass; see Phase 4’s post-phase note.

Phase 9 – Sniffer loader ✅

Game-protocol sniffers load into a running proxy from disk, sandboxed, never compiled in and never a fork. A sniffer: route resolves its name against the loaded set. The official sniffers (a2s, minecraft, regex_firstbytes, …) live in wayhouse-proxy/sniffers; the ABI crate they link is crates/wayhouse-sniffer-abi/.

Locked decisions

• Sandbox: wasmtime, core module, no WASI. A narrow ABI — the guest exports memory, alloc(u32) -> u32, and sniff(ptr: u32, len: u32) -> u64 (packed ptr<<32 | len; 0 = not recognised); the result bytes are a compact encoding of RouteHint. No WASI, no host functions ⇒ a sniffer cannot touch the filesystem, clock, or network.
• Time bound: epoch interruption. A host thread calls Engine::increment_epoch() every call_timeout_ms; a sniffer that overruns traps. (Fuel is optional on top.)
• Memory bound: StoreLimits max_memory + one instance per call, from settings.sniffers.max_memory_bytes.
• Host lives in the wayhouse binary. wasmtime is a binary-only dep, like reqwest / hickory-resolver; the Sniffer trait and the MatchContext.sniff wiring stay in wayhouse-core (crate-boundary rule 7).
• Registry threading. The free fn sniff::sniffer(name) (global, 'static) becomes an Arc<Sniffers> map (HashMap<String, Arc<dyn Sniffer>>) threaded runtime → ListenerManager → workers, exactly like Arc<Resolvers> / Option<Arc<GeoDb>>. This is the one real refactor.
• First-party sniffers ship built, the same way: a2s, minecraft, and a generic bounded regex-firstbytes (keeps regex off the core routing path). sni stays a native Matcher — it is not a sniffer.
• Latency is a gate, not an assumption. Bench the WASM boundary vs. NFR N1 (< 0.5 ms added) before wiring it on the per-connection path. Fallbacks, in order: InstancePre + a pooling allocator; one warm instance per worker reset between calls; ultimately the out-of-process Phase-4 resolver.

Config (settings.sniffers)

dir (*.wasm, loaded at startup, rescanned on reload), call_timeout_ms, max_memory_bytes, optional modules: [{ name, sha256 }] for supply-chain pinning. Per-listener sniffer: is unchanged; the input is already capped at the route’s peek_len() ≤ PEEK_MAX.

Slices (all done — see HANDOVER.md codebase map + git history for detail)

• ✅ Slice 1: registry threading (pure refactor) — sniff::sniffer global fn → wayhouse_core::sniff::Sniffers, an Arc<Sniffers> threaded to every listener worker like Arc<Resolvers> / Option<Arc<GeoDb>>. No built-in sniffers register in production; no wasmtime yet.

• ✅ Slice 2: settings.sniffers schema (dir, call_timeout_ms, max_memory_bytes, modules: [{ name, sha256 }]) → wayhouse_config::SniffersConfig. Config-only — nothing reads it yet.

• ✅ Slice 3: WasmSniffer (crates/wayhouse/src/sniffer_loader.rs) — a shared wasmtime::Engine (epoch interruption) + epoch-ticker thread, a fresh Store/Instance per call under a StoreLimits memory cap and a one-tick epoch deadline. ABI: guest exports memory, alloc(len) -> ptr, sniff(ptr,len) -> packed|0 (later widened by ADR 16a); host decodes a compact RouteHint — any bad pointer/length/UTF-8 is bad_output, never a panic. build_sniffers(&SniffersConfig) scans dir, verifies sha256 pins, fails closed on a bad sniffer (like geo_db). wayhouse_sniffer_calls_total{name,result} / wayhouse_sniffer_call_seconds{name}.

• ✅ Slice 4: live dir rescanning on reload — Sniffers gained ArcSwap interior mutability (mirrors RouteHints) so replace() from the reload task is visible to every worker instantly; a scan error keeps the previous sniffer set. Engine params stay startup-only.

• ✅ Slice 5: first-party sniffer crates (built at the time in a standalone sniffers workspace in this repo, moved to wayhouse-proxy/sniffers in #183; never a dep of wayhouse/wayhouse-core) — wayhouse-sniffer-abi (guest-side ABI helper) + a2s (Source-engine query packets), minecraft (Handshake server address, the BungeeCord/Velocity virtual-host trick), regex-firstbytes (a bounded pattern-matcher template, later made genuinely configurable by ADR 16a). Now: make sniffers-fetch; CI job sniffers-e2e.

• ✅ Slice 6: WASM-boundary latency bench vs. N1 (sniffer_loader::tests::wasm_boundary_latency_vs_nfr_n1, #[ignore]d, run in the sniffers CI job). Result: p50 8–10 µs, p99 12–26 µs across all three sniffers — comfortably inside N1, so the fresh-Store-per-call design needs none of the InstancePre / warm-instance fallbacks held in reserve. Found along the way: the epoch ticker runs on wall-clock time for the loader’s whole lifetime, so call_timeout_ms is a ceiling, not a per-call guarantee — documented in docs/07 “Sniffer sandbox guarantees”.

• ✅ Slice 7 — phase 9 complete: end-to-end test driving a live Runtime

  • real TCP connection through the real build_sniffers directory scan (a WAT fixture compiled inline, no wasm toolchain needed) — not #[ignore]d, runs in every make check.

• Result — phase 9 complete: runtime-loaded, sandboxed game-protocol sniffers (a2s, minecraft) plus a regex-firstbytes template, all measured inside NFR N1; regex first-bytes matching lives in an optional sniffer, never in core. Known follow-ups, not blocking: per-source cap LRU eviction — see HANDOVER.md. (GET /sessions since landed — data-plane completion.)

• ✅ Post-phase-9 (data-plane completion): per-sniffer config. ADR 16a widens the guest ABI to sniff(in_ptr, in_len, cfg_ptr, cfg_len); settings.sniffers.modules[] grows a config string, marshalled into a second linear-memory region on every call. a2s / minecraft ignore it; regex-firstbytes was rebuilt around it — a tiny [key:NAME|] [@OFFSET ](hex:…|ascii:…){|…} pattern language, so it is a genuinely runtime-configured bounded matcher and not the hard-coded HTTP template.

Phase 10+11 (merged) – PoC: single controller + single aggregator ✅

Full design: 10-distributed-control-plane.md (“Fleet topology”, “The controller”, “The aggregator”, “The admin GUI”). This is the first working release of the two new services, deliberately scoped down from the full design: one standalone controller tier, one aggregator tier, replicas: 1 on each, no slave role, no HA, no adoption flow. Those are real, already-designed extensions (see docs/10) but are explicitly a later release, not part of this one — ship something that works end-to-end first. No change to the hot path or the Snapshot invariant in either slice group.

Scope call locked for this release: operator intent (backend overlay/admin-state, route hints) stays exactly as it is today — per-instance, via the existing admin API — and does not move into the controller yet. The controller in this release owns structural config only. The aggregator’s intent fan-out (below) is a thin proxy to each instance’s existing admin verbs, not a new durable intent store. Moving intent into the controller’s revision log is a follow-on once this ships.

Controller slices (wayhouse-controller, structural config distribution)

1. ✅ Skeleton binary + store::Store — Tier-1 store as an embedded KV (sled, ADR 20), single node, no Raft/etcd for this release. open / current_revision / get / current / put; put assigns the next monotonic revision and persists it + the current pointer in one sled transaction, then flushes. wayhouse-controller binary opens the store and serves GET /healthz (same observability floor as wayhouse). Nothing calls POST /config yet — that’s slice 2.
2. ✅ POST /config: runs the same wayhouse_config::parse_str (parse + validate()) a proxy runs on a file reload, then Store::put. Rejects (422, error body) and leaves the current revision untouched on an invalid submission. GET /config returns the current revision’s raw text
   • an X-Config-Revision header (404 before the first submission).
3. ✅ GET /config/subscribe?since=<revision>: SSE. Sends the catch-up range (Store::revisions_after(since)) then tails a broadcast::Sender<u64> fed by submit_config; a lagging subscriber (a RecvError::Lagged) just re-runs the catch-up query from wherever it left off — Store never forgets a revision, so there is no delivery state on the writer side (docs/10 principle 5). 5 new tests, incl. one that forces a lag and confirms no revision is skipped.
4. ✅ wayhouse gains --controller <url> (mutually exclusive with --config via clap conflicts_with). controller_client::fetch_current does the initial GET /config (mirrors wayhouse_config::load in file mode); controller_client::run then holds the subscribe connection and feeds every accepted revision through a new shared reload::apply_config (the rebuild/reconcile tail reload::apply and the controller path both call — only how the Config was obtained differs). On disconnect: reconnect with capped exponential backoff (500 ms → 30 s) from the last-applied cursor; the last-applied snapshot keeps running the whole time (freeze-on-last-known-good, never clear — verified live: killing the controller mid-session left the proxy serving traffic on its last config while retrying). An invalid pushed revision is logged and skipped (cursor still advances — it must not be replayed forever on every reconnect), the same “bad reload keeps the old snapshot” rule as a file reload. --controller-token (added later, slice 11f testing) is what wayhouse presents to a --auth-token-protected controller — every other cross-service link in this fleet has a token pairing, and this one was missing until driving the real UI against a real fleet surfaced it.
5. ✅ GET /config/revisions (history: revision, size, current flag), GET /config/revisions/{revision} (a past revision’s raw text), GET /config/revisions/{revision}/diff[?against=<revision>] (a similar-crate line diff against current or another revision, +/-/ -prefixed plain text), POST /config/rollback/{revision} (never rewrites history — re-submits that revision’s bytes through the same validate-then-Store::put path as POST /config, so it’s just an ordinary new revision to every subscriber, no special-casing anywhere else). --auth-token <token> on wayhouse-controller gates the whole /config* surface with a bearer-token check (/healthz stays open); a shared secret, not RBAC — appropriate for this release’s one-controller scope. 8 new tests; verified live end-to-end over real HTTP (401 without the token, history/diff/rollback all round-tripped against config.example.yaml).

Aggregator slices (wayhouse-aggregator, fleet view + operational verbs)

6. ✅ Skeleton binary (new crate, no wayhouse-core/wayhouse-config dependency — the aggregator stays fully decoupled from the data-plane crates) + POST /ingest. ingest::IngestStore: an in-memory, latest-write-wins map keyed by self-reported instance — deliberately never persisted, see the “stateless and ephemeral by design” note in the crate’s lib.rs (an aggregator restart loses nothing that isn’t about to be re-pushed on the next tick). Payload is a summary (pool/backend health + admin state, session counts), not the full live session registry — that already exists per-instance (GET /sessions), break-glass style. 12 new tests; verified live over real HTTP.
7. ✅ wayhouse gains --aggregator <url> (+ --aggregator-instance, --aggregator-interval-sec, default 10s) — independent of --controller, pushing state and pulling config are unrelated axes. aggregator_client builds an IngestPayload straight from the live RuntimeHandle (snapshot().pools for pool/backend summaries, sessions() filtered by Proto for TCP/UDP counts) and POSTs it every tick; the wire shape is duplicated in wayhouse rather than adding wayhouse-aggregator as a dependency (same reasoning as controller_client hand-parsing the controller’s SSE JSON instead of depending on wayhouse-controller). No retry buffer — a deliberate deviation from this line’s original wording: IngestStore is latest-write-wins state, not an event log, so replaying an old failed push after a fresher one already landed would make the aggregator’s view older; a failed push just logs and is superseded by the next tick’s fresher snapshot. (Push, not pull — see “Fleet topology” in docs/10 for why: no inbound network path to a proxy’s admin port is ever needed, at any deployment size.) 1 new test (build_payload against a real Runtime); verified live end-to-end over real HTTP (WAYHOUSE_LOG=debug showed two successful pushes 2s apart, matching --aggregator-interval-sec 2).
8. ✅ GET /fleet/pools / /fleet/sessions / /fleet/healthz — served directly from IngestStore (no fan-out RPC needed, the data already arrived), each entry carrying last_seen_ms_ago (the aggregator’s own clock). /fleet/healthz flags an instance stale past 30s (~3x wayhouse’s default push interval) — the threshold IngestStore itself deliberately doesn’t have. No GET /fleet/config in this release: IngestPayload is state, not config content — the controller already owns that (GET /config//config/revisions). A fleet-wide “which revision is each instance running” view is real and useful (an optional config_revision field on IngestPayload) but needs controller_client and aggregator_client to share state inside wayhouse that today are deliberately independent — deferred, not dropped. 8 new tests (incl. a test-only IngestStore::insert_state seam to test the staleness threshold without a real 30s wait); verified live end-to-end over real HTTP.
9. ✅ Intent-verb fan-out (wayhouse-aggregator/src/fanout.rs), thin and stateless — the aggregator decides nothing, stores no intent, just relays using each instance’s self-reported admin_url (a new IngestPayload field, http://{settings.admin.listen}). Two shapes: targeted (POST /fleet/instances/{instance}/drain|undrain — draining an instance only ever means one instance, so this passes that instance’s own response straight through, 404 for an unknown name) and broadcast (POST /fleet/pools/{pool}/backends, PATCH/DELETE .../backends/{addr}, POST /fleet/route-hint — no shared owner across instances this release, so applying fleet-wide means calling every known instance’s own admin API independently via a tokio::task::JoinSet, reporting per-instance results; one bad instance never fails the others). 9 new tests using real mock instance HTTP servers (reachable, unreachable, and a real Json extractor to catch content-negotiation bugs) — one of them (a_broadcast_sets_content_type_even_if_the_caller_ never_did) is a regression test for a real bug the live smoke test caught: the broadcast forwarded a caller’s body without ever setting Content-Type, so a caller that omitted it got 415 from the target’s own Json extractor even though the body was valid JSON — fixed by setting application/json explicitly on every forwarded body, since these endpoints are always JSON regardless of what the caller remembered to send. Verified live end-to-end over real HTTP (targeted drain flipped the real instance’s /readyz to 503; broadcast add-backend landed on the instance’s real /pools).
10. ✅ Bearer-token auth, three independent secrets for three independent hops (never one token threaded through everything): wayhouse-aggregator --auth-token gates its own API (/ingest, /fleet/*, fan-out writes; /healthz stays open); settings.admin.auth_token (new wayhouse-config field) closes the “admin API has zero auth” gap on wayhouse’s own admin API the same way (admin.rs gains a require_bearer middleware, mirroring wayhouse-controller’s); wayhouse --aggregator-token is what a proxy presents pushing to /ingest; wayhouse-aggregator --instance-token is the separate secret the aggregator presents out to every instance’s admin API when fanning out (settings.admin.auth_token on the instance side must match). 3 new tests (admin auth gate, aggregator auth gate, wayhouse-config parsing auth_token). Verified live end-to-end over real HTTP through every hop: unauthenticated aggregator read → 401; authenticated read showed the real pushed data; unauthenticated instance admin call → 401; a fan-out drain (through the aggregator, correctly presenting --instance-token) → 200, confirmed real by the instance’s own /readyz flipping to 503.

All 10 slices of the phase 10+11 PoC’s controller + aggregator halves are now done — the remaining slices (11–13) are the Web UI, integration tests, and docs polish.

Web UI + tests

Slice 11 is a dedicated wayhouse-ui process — a BFF (backend-for-frontend), not a static SPA calling the controller/aggregator bearer-token APIs directly. Design refinement made when actually starting it, now locked in docs/10 (“The admin GUI”): the browser gets its own session-cookie login on wayhouse-ui, wholly separate from every machine-to-machine bearer token (--auth-token/--instance-token/--aggregator-token/ settings.admin.auth_token), none of which a browser ever holds — wayhouse-ui holds the controller’s and the aggregator’s tokens itself and calls both on the operator’s behalf. wayhouse-ui has no store and no fleet data of its own (nothing outlives a restart beyond active sessions) — it is not a third authority, it authorizes nothing itself beyond “is this a valid session.” A React + Vite + TypeScript SPA it serves itself (tower-http::ServeDir), one process, one port for the operator. No proxy admin port, and no machine token, is ever exposed to a human directly.

11a. ✅ wayhouse-aggregator gains GET /fleet/subscribe (SSE, bearer-gated like the rest of /fleet/*) — the machine-to-machine feed wayhouse-ui (11d) subscribes to for live updates: current merged fleet state (FleetInstanceView — pools + sessions + a stale flag, one entry per instance) on connect, then a resend, debounced 150 ms against a burst, on every accepted POST /ingest. Reuses the exact catch-up-then-broadcast shape wayhouse-controller’s /config/subscribe already proved out, applied to state instead of a revision log — no cursor/replay semantics needed, a Lagged subscriber just means “rebuild now,” same as any other signal. 3 new tests (immediate initial send, resend-after-debounce, a 5-push burst collapsing into exactly one resend); verified live over real HTTP against a real wayhouse push loop. 11b. ✅ New crate wayhouse-ui (lib + bin, no wayhouse-core/wayhouse-config dependency — stays as decoupled as wayhouse-aggregator is). --ui-password: POST /ui/login {password} issues a random 256-bit session id (session::SessionStore, in-memory — a restart just logs everyone out, the same “ephemeral, nothing durable” posture the aggregator already has), returned as an HttpOnly, SameSite=Lax cookie (not yet marked Secure — noted as a gap for a TLS-fronted deployment, not silently ignored; since 2026-10-02 a UI serving native TLS, --tls-cert, sets Secure itself, behind a proxy the proxy must); POST /ui/logout clears it; GET /ui/session (gated by the new require_session middleware) lets the frontend check login state on load. None (--ui-password omitted) leaves the UI open, consistent with every other optional-auth surface in this fleet. 9 new tests; verified live end-to-end over real HTTP through the whole cycle: unauthenticated → 401, wrong password → 401, right password → a cookie that unlocks the gated route, logout → 401 again. 11c. ✅ wayhouse-ui proxies reads + slice-9 operational verbs to wayhouse-aggregator (--aggregator-url/--aggregator-token) — a new aggregator_proxy.rs, thin and stateless like wayhouse-aggregator::fanout it calls through to: GET /api/fleet/pools/sessions/healthz, POST /api/fleet/instances/{instance}/drain|undrain, backend add/patch/delete, route-hint. Translates the browser’s session cookie into the aggregator’s bearer token server-side — the browser never holds it. 503 if no --aggregator-url was configured at all, 502 if it was but couldn’t be reached, both handled cleanly (no panic). This is phase 10’s “operational” GUI capability level (docs/10): read-heavy plus these verbs, no structural config editing yet. 6 new tests (incl. a real mock-aggregator round-trip proving the token and request body are forwarded correctly). Verified live end-to-end over the real four-hop chain: unauthenticated browser call → 401; logged in → GET /api/fleet/pools showed the real pushed data; a drain through the full wayhouse-ui → wayhouse-aggregator → wayhouse path landed for real, confirmed by the instance’s own /readyz flipping to 503. 11d. ✅ wayhouse-ui’s GET /ws/fleet: a WebSocket to the browser, fed by a single shared crate::fleet_feed subscription to wayhouse-aggregator’s slice-11a SSE feed (one aggregator connection total, fanned out to every browser tab — not one per tab). fleet_feed::run reuses wayhouse’s own controller_client’s hand-rolled SSE parsing (a chunked-body loop splitting on blank lines) and reconnects with the same capped exponential backoff on disconnect; a latest cache means a browser connecting between two pushes gets the current view immediately rather than waiting for the next one. A Lagged WS subscriber resends latest rather than replaying — same “this is state, not a log” reasoning as wayhouse-aggregator’s own subscribe_fleet_worker. Gated by require_session like everything else the browser reaches (a WS upgrade is an ordinary GET until the 101 handshake). 5 new tests, 3 of them against a real WebSocket client (tokio-tungstenite) and a real axum::serve listener, not mocks. Verified live end-to-end over the full five-hop chain with a throwaway probe client: real wayhouse → real wayhouse-aggregator (SSE) → real wayhouse-ui (fleet_feed) → a WebSocket client, watching live pushes arrive in real time. 11e. ✅ wayhouse-ui proxies the controller’s config API too (--controller-url/--controller-token, new controller_proxy.rs): GET/POST /api/config, GET /api/config/revisions(+/{rev}(/diff)), POST /api/config/rollback/{rev} — phase 10’s “full management” GUI level, straightforward since wayhouse-ui already holds a separate token per service. The controller’s POST /config body is raw YAML text (its handler takes a plain String, not JSON), forwarded byte-for-byte with no content-type forced on it, unlike the JSON bodies aggregator_proxy forwards. The diff route forwards the incoming query string (?against=) via axum::extract::RawQuery onto the proxied URL. 6 new tests. Found and fixed a real bug along the way: both proxy helpers (here and in aggregator_proxy, plus wayhouse-aggregator’s own fanout::proxy_to_instance) forwarded only status + body, silently dropping every response header — caught immediately by a test asserting X-Config-Revision survived the hop, which it didn’t. Fixed with a shared proxy_util::forwardable_headers (strips only the hop-by-hop headers connection/transfer-encoding/ content-length, forwards everything else verbatim) and a matching fix + regression test in wayhouse-aggregator. Verified live end-to-end over the full chain against a real wayhouse-controller: submit → list revisions → diff → rollback → GET /api/config correctly showing X-Config-Revision: 3 (the new post-rollback revision, not a rewind). 11f. ✅ Frontend: React + Vite + TypeScript in crates/wayhouse-ui/web/ (own package.json, never a Cargo workspace member — same reasoning as crates/wayhouse-config/fuzz/; make ui runs npm install && npm run build, output goes to dist/). wayhouse-ui gained --static-dir (default crates/wayhouse-ui/web/dist), served as a fallback under whatever the API routes don’t claim, via tower-http::services::ServeDir (new dependency). No client-side routing — one page, view state (which tab) lives in React state, not the URL; nothing here needs a deep link yet. Login (session-cookie form), FleetView (per-instance/pool/backend table live over 11d’s WebSocket, drain/undrain, backend add/patch/ remove, route-hint — phase 10’s “operational” level), ConfigView (editor + submit + revision history/diff/rollback — phase 10’s “full management” level). src/api.ts holds every fetch call; wayhouse-ui’s session cookie is the only credential the browser ever sends — never a bearer token, per the whole point of the wayhouse-ui redesign. Verified live end-to-end with the real built frontend served by a real four-process fleet (wayhouse + wayhouse-controller + wayhouse-aggregator + wayhouse-ui, all running together for the first time): index.html and a JS asset served with correct content-type, login, a config submission and a fleet-pools read through the served UI’s own proxy paths, and a drain issued through the full chain landing for real (confirmed by the instance’s own /readyz flipping to 503).

• Slice 11 (all of 11a-11f) is now complete. The whole wayhouse-ui BFF — session login, fleet reads/ops proxying, the live WebSocket, config editing/history proxying, and the actual frontend serving all of it — has been verified live end-to-end, repeatedly, against real running wayhouse/wayhouse-controller/wayhouse-aggregator processes, not just against unit tests.
• The 11f frontend itself was a functional PoC, not a finished operator UI (confirmed by the user actually clicking through it in a browser; since redesigned — see “Later / optional” below) — every backend path it drives is real and solid, but the UI layer is deliberately bare: no styling/design pass, no confirmation dialogs before a destructive action (drain, remove-backend fire immediately), no loading states beyond a bare “loading…”, no client-side routing, plain unstyled tables and forms. A real frontend overhaul is future, separate work — tracked under “Later / optional” below — deliberately deferred rather than polished now, since it doesn’t block anything else in this phase.

12. ✅ Integration tests: N wayhouse instances + 1 controller + 1 aggregator — subscribe/reconnect/freeze-on-disconnect, push/ingest, fan-out partial failure, config reject-keeps-previous. New crate crates/wayhouse-fleet-tests (workspace member, part of make check): spawns the real wayhouse/wayhouse-controller/wayhouse-aggregator binaries as child processes on loopback with OS-assigned ports and drives them over real HTTP, rather than an in-process harness — matching how every earlier slice was actually verified (see HANDOVER.md) and specifically able to catch the wire-shape class of bug slices 11e/11f already found live. 4 tests, all green. wayhouse-ui isn’t spawned here — nothing in the 4 listed scenarios exercises it, and it has no state of its own to assert on beyond what slice 11’s own tests already cover.
13. ✅ Docs: docs/06 gained a “Fleet control plane” section (full endpoint reference for wayhouse-controller/wayhouse-aggregator/wayhouse-ui, and updated the now-stale “not persisted or fleet-synced today” / “no built-in fleet-wide view” callouts in “Multi-instance operations” to point at it); README.md status block now covers phase 10+11; this status legend; HANDOVER.md.

• Result: one controller to distribute config to a fleet, persistently and with an audit trail, and one aggregator to view and operate that fleet from one screen — a real, working v2 release. The standalone/slave hierarchy, intra-tier HA, adoption, and moving intent into the controller are the next release on top of this, not blocking it.
• All 13 slices of phase 10+11 are now complete.

Phase 12 – Fleet hierarchy, HA, and shared intent ✅

Full design: 10-distributed-control-plane.md (“Fleet topology”, “Adoption”). Builds on phase 10+11’s single-tier PoC — additive, no rework of what shipped there.

• ✅ Slice 1 (wayhouse-controller role): standalone / slave role, static and install-time (--role, never inferred from connectivity). A standalone tier is unchanged from phase 10+11. A slave tier (--parent-url + --parent-token) never accepts a write directly (POST /config and /config/rollback/* both 403) — instead parent_client seeds from the parent’s GET /config at startup, then subscribes to GET /config/subscribe and relays every accepted revision into its own store (its own local revision numbers, not the parent’s — only the payload shape is shared). Reuses wayhouse’s own controller_client’s reconnect-with-backoff shape; a lost parent freezes the slave on last-known-good and keeps serving/relaying it downward, same “never clear” rule as the proxy-to-controller hop. Verified live: root submits a revision → slave relays and serves it with its own X-Config-Revision; a direct write to the slave is 403; killing the root leaves the slave serving what it had and retrying with backoff. Not yet built at the time: aggregator-side hierarchy, intra-tier HA (Raft/etcd), intent migration into the revision log, RBAC, canary rollout, adoption.
• ✅ Slice 2 (wayhouse-aggregator hierarchy): --parent-url (+ --tier-name, --parent-token, --parent-push-interval-sec) makes a wayhouse-aggregator tier also push its own merged view up to a parent aggregator’s POST /ingest, on a fixed interval, via new parent_push.rs — “a tier’s aggregator is itself a valid leaf to its parent aggregator” (docs/10). Deliberately namespaces rather than merges: every instance this tier currently knows is pushed individually with its instance renamed "{tier_name}/{instance}", so the parent’s IngestStore//fleet/* need no new payload shape or special-casing — it just sees more, longer-named instances (namespacing only touches the copy sent upward; this tier’s own local names and /fleet/* view are untouched). 3 new tests, incl. one against a real mock parent axum::serve listener. Verified live: a child aggregator ingesting proxy-1 shows up on a real parent aggregator’s GET /fleet/sessions as region-a/proxy-1. Not yet built at the time: intra-tier HA, intent migration, RBAC, canary rollout, adoption.
• ✅ Slice 3 (operator intent → the controller’s revision log): pool-scoped operator intent — backend add/remove, backend admin state, route-hint — now has a fleet-wide, persisted path through wayhouse-controller, alongside (not replacing) direct per-instance admin API calls. New wayhouse-controller::intent module: a second sled log (its own <data_dir>/intent database, reusing Store unchanged), POST /intent (validates the op’s shape — addr/IP parse, known state — before it’s ever broadcast, since there’s no wayhouse_config::validate() to lean on for a bare op) and GET /intent/subscribe, the identical catch-up-then-tail shape config uses. Deliberately excludes whole-instance drain/undrain (it targets one instance, not “every instance with pool X” — stays on the aggregator/direct-admin path) and resolver pins (still an open question, docs/01). New wayhouse module intent_client.rs subscribes whenever --controller is set and applies each op through the exact same RuntimeHandle calls crate::admin’s handlers make — an intent op is a new source for an existing mutation, not a new code path. 15 new tests across both crates. Real pre-existing bug found and fixed along the way, via this slice’s live smoke test: in --controller mode, nothing was watching RuntimeHandle::reload_requested() at all — reload::run is the only thing that ever did, and it’s only spawned in file-config mode — so the phase-5 admin API’s own POST/DELETE /pools/{pool}/backends (backend overlay edits) silently never took effect for any --controller instance, before this slice existed. Fixed with a new controller_client::watch_admin_reloads, spawned alongside --controller: debounces request_reload() notifications and re-fetches
  • re-applies the controller’s current config (rather than caching a local copy, so there’s nothing to drift from the controller’s own view). Verified live end-to-end: wayhouse-controller + wayhouse --controller running for real, a POST /intent backend_add landed in wayhouse’s own GET /pools output within one debounce window. Not yet built at the time: intra-tier HA, slave-tier intent relay (only config relayed through a parent then), RBAC, canary rollout, adoption.
• ✅ Slice 4 (slave-tier intent relay): the intent-log counterpart to slice 1’s config relay. New wayhouse-controller::intent::relay, structurally identical to parent_client (subscribe-with-backoff, land via IntentState::apply_revision — the same role-gate bypass) with one difference: no fetch_initial seed step, since an intent log has no single “current document” to seed from — since=0’s catch-up on the first subscribe already replays the parent’s whole history, unlike config’s “must serve something the moment a fresh slave comes up.” A slave tier now relays both logs from its parent, still never originating either directly (403 on both /config and /intent writes). 4 new tests. Verified live with two real wayhouse-controller processes: root submits an intent op → slave relays and serves it under its own local revision number; direct writes to the slave still 403; killing the root freezes the slave on both logs, retrying with backoff. Not yet built at the time: intra-tier HA, RBAC, canary rollout, adoption.
• ✅ Slice 5 (adoption): POST /admin/adopt {"parent_url":...} flips a running standalone tier to slave without a restart (docs/10 “Adoption”). Solves the design doc’s two open correctness questions the same simple way: adoption is refused (409) unless both the config and intent stores are still empty — a tier with its own revision history must be replaced by a fresh one to join a hierarchy, not reconciled in place; an empty store trivially has nothing in flight to reorder either. New shared role::RoleHandle (an Arc<RwLock<Role>>, replacing the plain Role field AppState/IntentState held) so the flip is visible to both write gates instantly, from every clone. On success: seeds from the new parent’s config (mirrors main.rs’s own boot-time seed) and spawns the same parent_client/intent::relay tasks a --role slave boot would have — a freshly-adopted tier is indistinguishable from one that started as a slave. 6 new tests. Verified live with two real wayhouse-controller processes: adopting a fresh child returned {"seeded_config_revision":1}, its GET /config immediately served the root’s config, and a direct write to it was 403 from that point on. Still not built: intra-tier HA, RBAC, canary rollout.

Slices 6–8 (2026-09-05 design session, now all built) — each has its own docs/10 section (with wire shapes, storage layout, and rejected alternatives) and a docs/09 ADR (21–23):

• ✅ Slice 6 — Intra-tier HA (wayhouse-controller only; the aggregator’s HA design — stateless replicas behind one address, no consensus needed — stays design-only, not built): embedded openraft 0.9, one Raft group per controller tier replicating both the config and intent logs. New wayhouse_controller::ha module: a sled-backed LogStore (RaftLogStorage/RaftLogReader, its own <data_dir>/ha database — durable across a restart, unlike openraft‘s own in-memory reference implementation this was adapted from), a StateMachineStore (RaftStateMachine/RaftSnapshotBuilder) that applies a committed entry by calling the exact same AppState/IntentState::apply_revision a direct write already used, a reqwest-based RaftNetwork posting to peers’ /raft/* routes (gated by a new peer-only --ha-token), and ha::client::propose_write — the one call api::submit/ intent::api::submit_intent make instead of apply_revision directly when --ha-peers is set: proposes via Raft, and transparently HTTP-forwards to the current leader (never a redirect) if this replica isn’t it, so no client anywhere needs to know HA exists. New CLI: --ha-node-id, --ha-peers id=host:port,... (identical on every replica; each boots and calls raft.initialize() with the same static set — harmless no-op on every node but the one that wins the race), --ha-token. Scope cut at the time, built since (2026-10-05): HA and the slave role were mutually exclusive in this slice; the leader-only upward relay with a replicated cursor, designed in docs/10, now exists. 16 new tests. Verified live with a real 3-node cluster: booted nodes 1/2/3, confirmed node 3 elected leader; submitted a config write to node 1 (not the leader) — transparently forwarded, 200, {"revision":1}; submitted to node 2 — {"revision":2}; all three nodes’ GET /config agreed on revision 2. Then killed the leader (node 3) and confirmed real failover: node 1 was elected the new leader within seconds, and both a config write (revision 3) and an intent write both succeeded through the surviving 2-node majority.
• ✅ Slice 7 — Staged / canary rollout: one monotonic config log, never forked — revisions tagged {promoted, canary_groups} in a new stage sled tree (a sibling of Store’s own trees, opened via a new Store::db() accessor; Store itself gains nothing, stays exactly the content-agnostic log it always was). POST /config?stage=canary&group= <name> submits a revision visible only to a subscriber reporting that group (GET /config?group=<name>, GET /config/subscribe?group= <name>); a plain POST/GET with no stage/group is byte-for-byte the pre-slice-7 behavior (every revision defaults Stage::promoted). POST /config/promote/{revision} is the one deliberate exception to “every change is a new revision” — flips an existing revision’s visibility in place. GET /config/revisions gained promoted/ canary_groups columns. subscribe_worker’s catch-up/tail now filter every candidate through Stage::visible_to, holding an invisible revision back (without advancing the subscriber’s cursor past it) until a later signal — its own promotion, or any newer submission — makes it either visible or moot. HA-aware: WriteRequest::Config now carries its Stage through the Raft log, and a new WriteRequest::Promote(revision) variant replicates a promotion the same way. Deliberately config-only, not the intent log — no clean partial-rollout meaning for a single already-narrow op (see docs/10). 7 new tests. Verified live: submitted a promoted v1 and a region-a-canary v2; a plain GET /config and GET /config?group=region-b both still showed v1; GET /config?group= region-a showed v2; POST /config/promote/2 then made v2 the plain GET /config answer too. See docs/10 “Staged / canary rollout (design)”, ADR 22.
• ✅ Slice 8 — RBAC and audit: multi-operator accounts (--users-file, argon2 PHC hashes, wayhouse-ui --hash-password to produce one) and three roles (viewer/operator/admin) enforced in wayhouse-ui — the controller/aggregator keep their existing single shared-token gates unchanged, per the design’s explicit continuation of the phase 10+11 slice-11 divergence. --ui-password is kept (not replaced) as a legacy single-shared-secret, implicitly-admin mode. New wayhouse_ui::role::Role (Viewer < Operator < Admin, derived Ord) and a SessionStore that now maps a session id to {role, username}; crate::auth::check_role(min) gates three separately-route_layered sub-routers (viewer/operator from aggregator_proxy, viewer/admin from controller_proxy) — 401 for no/invalid session, a new 403 for a valid session below the route’s minimum. A new X-Actor header (the session’s username, if any) rides every write wayhouse-ui proxies; wayhouse-controller records it per revision in a new actors sled tree (GET /config/revisions gained an actor field — through HA too: WriteRequest::Config now carries actor alongside stage, and leader-forwarding preserves the header so whichever node ends up proposing the write has it); wayhouse-aggregator forwards it to each instance and logs (instance/pool, actor, verb) via tracing (a convenience, not a durable record — this aggregator holds no durable state by design). 17 new tests across wayhouse-ui/wayhouse-controller/ wayhouse-aggregator. Verified live: hashed a password with --hash-password, logged in as that user against a real wayhouse-ui with --users-file, submitted a config through the full wayhouse-ui → real wayhouse-controller chain, and confirmed GET /config/revisions showed "actor":"alice" on the resulting revision. See docs/10 “RBAC and audit (design)”, ADR 23.

Phase 12 is now fully built — all 8 slices (hierarchy, both relay logs, adoption, intra-tier HA, staged/canary rollout, RBAC and audit) are implemented, individually verified live, and covered by make check. The design in docs/10 is realized: config/intent consistent and durable across an arbitrarily large, regionally structured fleet, no single point of failure at any tier, and per-role accountability for every change. Phase 13 (the regional health fabric, Tier 2) is now fully built — see below.

Phase 13 – Regional health fabric ✅

Full design: 10-distributed-control-plane.md (Tier 2). Advisory, rebuildable, off the data path. Fully built (2026-09-05) — all 5 slices below, verified live including two real multi-process tests in crates/wayhouse-fleet-tests; see “Mechanism (built)” in docs/10 and ADR 24 in docs/09 for the locked design this implements. Mechanism: membership via embedded foca (SWIM); per-backend health as a last-writer-wins (instance, backend) register piggybacked on foca’s own broadcast/anti-entropy; HMAC-SHA256 over a per-domain pre-shared key, not mTLS.

Slices:

1. ✅ Config schema (done): settings.failure_domain + settings.gossip {bind, seeds, quorum_fraction, psk} in wayhouse-config (raw + resolved types, validate() rejects one without the other), config.example.yaml, docs/05.
2. ✅ Membership (done): new wayhouse-core::gossip module wrapping foca::Foca (SWIM) over a plain UDP socket, spawned by Runtime::start_with_discovery only when settings.gossip is set (wayhouse/src/main.rs passes cfg.gossip.clone(); startup-only, like geo/sniffers — a reload does not start or stop the mesh). HMAC-SHA256-tagged/authenticated datagrams (bad or missing tag ⇒ dropped silently + wayhouse_gossip_auth_rejected_total); wayhouse_gossip_members / wayhouse_gossip_messages_total in metrics_defs.rs. This slice’s NoCustomBroadcast means no application payload rides the mesh yet — pure membership. 8 new tests (6 unit incl. two real two-process-equivalent in-tokio SWIM convergence tests; a wrong-PSK-never-joins test). Verified live with two real separate wayhouse processes on real sockets (not in-process): both converged to wayhouse_gossip_members 1 with real send/receive traffic on /metrics within the SWIM probe period.
3. ✅ Per-backend health broadcast (done): BackendHealthRegister {addr, up, changed_at, origin} is a last-writer-wins payload piggybacked via a new BroadcastMerger: foca::BroadcastHandler<SocketAddr> — foca’s own anti-entropy re-gossips it until fully propagated, no second sync pass. GossipHandle::publish_backend_health(addr, up) (an mpsc channel into the mesh task; the actual health.rs caller is slice 4) asserts this instance’s own verdict; GossipHandle::domain_votes(addr) / quorum_down(addr, fraction) read the merged view ((0, 0) votes ⇒ quorum_down is always false — an empty view never overrides, only a real quorum can). 5 new tests: pure domain_votes/quorum_down unit tests, and a real two-process-equivalent in-tokio test proving a published register actually reaches the other instance through foca’s own gossip (not a direct map share) and that quorum_down reads it back correctly. One real bug found by that test, not by inspection: an early draft merged the register into the domain map before calling foca.add_broadcast, which made foca’s own internal receive_item(data, None) call see an already-known key and silently skip queuing it for dissemination — fixed by letting add_broadcast perform the one and only merge (the sender: None case in BroadcastHandler::receive_item exists exactly for this “adding it myself” path).
4. ✅ Pool/health integration (done): Backend gained domain_down: AtomicBool alongside the existing healthy (untouched); is_healthy() = local_healthy() && !domain_down_flag(). New Backend:: observe_domain(quorum_down) can only set/clear domain_down, never healthy — a backend still only returns healthy on this instance’s own rise streak. health.rs::sweep gained an optional GossipFabric (handle + quorum_fraction, None when settings.gossip is unset): every active-check result also calls GossipHandle:: publish_backend_health(addr, ok) (only for backends this sweep itself just probed — the “only ever asserts what it checks itself” rule from the design), and after every sweep, every backend’s domain_down is refreshed from GossipHandle::quorum_down. Runtime::start_with_discovery builds one GossipFabric (if settings.gossip is set) and hands it to both the health task and the gossip task, so sweep never has to reach across tasks. wayhouse_backend_domain_down{pool,backend} gauge added. 5 new tests: 3 pure pool.rs unit tests (domain-down overrides but local rise alone clears it; domain-down clearing never revives a locally-down backend; domain_down carries across a reload by address, mirroring how healthy already does) + 2 health.rs tests, one proving sweep never touches domain_down with no gossip fabric, one driving a real single-node mesh through the full real pipeline (channel → mesh task → add_broadcast → merge) and confirming a repeated sweep eventually reads its own published verdict back and overrides the backend down.
5. ✅ Verification (done, crates/wayhouse-fleet-tests/tests/gossip.rs): two real multi-process tests, spawning actual wayhouse binaries the same way every other fleet slice was verified. domain_quorum_overrides_an_ instance_with_a_lenient_local_threshold gives 3 real processes the same real (unreachable) backend but a per-instance-different fall threshold (2 instances fall: 1, one fall: 1000) — a genuinely mixed local view achieved with plain config, no faked wire data — and confirms the lenient instance’s combined is_healthy() still goes false purely from the domain quorum, live proof of “an instance cannot ignore a domain-wide outage.” instances_with_different_psks_never_merge_and_ neither_gets_stuck confirms two real processes with mismatched PSKs never converge (wayhouse_gossip_members stays 0 on both) yet each still correctly runs its own local-only health checks and neither process hangs or crashes — the “fully rebuildable, falls back to local-only” property with a real empty mesh, not a config that merely looks empty.

• Each instance publishes its per-backend up | down; consumes the domain view.
• Health decision becomes quorum-weighted: unhealthy on local fall or domain quorum-down; healthy only on local rise; Tier-1 force-down overrides.
• Cold start / total partition ⇒ identical to today (own checks only).
• Result: faster, multi-vantage-point backend health across a domain; one bad vantage point no longer flaps a pool.

Phase 14 – Backend transport ✅

Full design: 11-backend-transport.md. Closes the gap docs/01-requirements.md’s Assumptions section states outright (“Backends are reachable over a trusted internal network”) — not true once proxy instances are distributed globally (phase 10–13’s whole point) in front of game servers on a different network entirely. Complete — all 7 slices built, see ADR 25 in docs/09.

Mechanism (design, locked 2026-09-05 follow-up session — see docs/11 “Locked decisions”): unmodified WireGuard via defguard/wireguard-rs (unifies kernel-netlink and boringtun-userspace config behind one API) as the tunnel data plane between an edge proxy and an origin — no new crypto/ framing protocol, and one shared interface per side with many peers, not one interface per origin. connect_backend/connect_upstream/ health.rs’s dials need no changes, since a tunneled backend is just a routable SocketAddr once the interface exists. An origin is modeled as a new BackendSource (ADR 12a) — backend_sources[].type: tunnel — not a new schema concept. The two genuinely new pieces: a small origin-side wayhouse-agent crate that manages the local WireGuard interface and registers its pubkey/backend addresses, and a “backend peers” registry on the existing Tier-1 wayhouse-controller that distributes peer configuration to every subscribed edge proxy the same way config revisions already are. WireGuard’s own roaming + keepalive mean only the proxy side ever needs a public endpoint — an origin behind a home NAT needs no port forwarding.

Slices, ordered by dependency (each was buildable and testable before the next needed it; all now built):

1. ✅ Config schema: backend_sources[].type: tunnel (wayhouse-config raw + resolved types, validate(), config.example.yaml, docs/05) — parses and validates with no runtime behavior behind it yet, same “schema first, mechanism after” shape phase 13 slice 1 used. Fields: at minimum the origin’s expected public key and which pool(s) it feeds; exact shape decided in this slice, not locked in docs/11.

2. ✅ wayhouse-controller’s backend-peers registry: a new resource alongside the config-revision and intent logs (its own sled tree/DB, matching Store’s own precedent) — POST for an agent to register (pubkey, allowed backend addresses, last-known endpoint) and a subscribe endpoint mirroring GET /config/subscribe’s catch-up-then- tail shape. Buildable and independently testable with nothing but curl/the crate’s own HTTP tests — no real WireGuard, wayhouse-agent, or wayhouse integration needed yet, same as how the config/intent logs were each built and tested standalone before any client integrated with them.

3. ✅ New wayhouse-agent crate: creates/maintains one local WireGuard interface via defguard/wireguard-rs, registers with slice 2’s registry on startup and on change. Verifiable standalone: point a real wayhouse-agent at a real wayhouse-controller, confirm its registration lands and a real local WireGuard interface comes up with the right key.

4. ✅ wayhouse’s subscribe-and-reconcile task: mirrors controller_client.rs’s shape, subscribes to slice 2’s registry, and reconciles the proxy’s own shared WireGuard interface’s peer list (via wireguard-rs) to match. Verifiable standalone against slice 3’s real agent: the proxy’s interface should show the agent as a peer within one subscribe cycle, independent of any actual game traffic yet.

5. ✅ The new tunnel BackendSource (wayhouse binary, same seam as the existing DNS-SRV/Consul/Kubernetes sources): fetch() resolves an origin’s currently-registered backend address(es) from the same peers registry slice 4 already subscribes to. This is what actually lets a pool’s source: <origin> produce live, tunnel-internal backend addresses through the existing discovery reconcile path — no changes needed to Snapshot::build_with_sources itself.

6. ✅ End-to-end live verification: a real wayhouse-controller + wayhouse-agent (+ a plain TCP echo backend) + wayhouse proxy in four Docker containers (NET_ADMIN + /dev/net/tun on the two WireGuard sides — this project’s own dev sandbox has neither, so containers were the practical way to get a CAP_NET_ADMIN-capable host), a real payload round-tripped through client → wayhouse’s public listener → WireGuard tunnel (boringtun userspace backend) → wayhouse-agent’s interface → the echo backend → back, confirmed stable (a live handshake surviving many re-registration cycles, not just a one-shot connect) — see HANDOVER.md for the two real bugs this surfaced and fixed: wayhouse-agent never added the edge proxy as a peer at all (closed with --peer-pubkey/ --peer-endpoint), and tunnel_client::run tore down and rebuilt the WireGuard session on every re-registration even when nothing changed (boringtun panics on a same-pubkey configure_peer, and the constant churn never let a handshake stabilize) — fixed by skipping a registration identical to the last one applied.

7. ✅ Proxy-peers registry — the mirror-image direction: slice 6’s end-to-end verification shipped wayhouse-agent --peer-pubkey/ --peer-endpoint as a static single-proxy pin, which proved the tunnel data plane but doesn’t scale — a growing proxy fleet, or a proxy added after an origin was already deployed, needed the origin’s wayhouse-agent restarted with new flags to learn about it. Closed by mirroring slices 2+4’s mechanism in the other direction: a new wayhouse-controller “proxy peers” registry (wayhouse_controller::proxy_peers, POST/GET /proxy-peers(+/{name}), GET /proxy-peers/subscribe) that every wayhouse --tunnel-* instance registers itself into (new wayhouse:: proxy_register, mirrors wayhouse-agent::register), and a new wayhouse-agent subscribe-and-reconcile task (proxy_subscribe.rs, mirrors wayhouse’s own tunnel_client.rs, including the same remove-then-add + skip-unchanged fix slice 6 needed) that reconciles every registered proxy onto the origin’s interface. --peer-pubkey/--peer-endpoint still work alongside it as a manual pin — the two converge to the same state, no conflict. Live-verified (2026-10-01) by make tunnel-e2e scenario 3: a proxy started after the origin is learned by the already-running agent with no restart, and carries traffic. That run also found a real bug: every proxy peer got AllowedIPs 0.0.0.0/0, so a second proxy stole the first one’s route. Fixed 2026-10-02 with the tunnel address authority (docs/11 “Address authority”): peers are /32s, and two_proxies_share_one_origin now passes in the lab.

Still open (see docs/11 “Open questions”): the CGNAT/both-sides-restrictive- NAT fallback (documented v1 limitation, no code) — does not block starting slice 1.

Later / optional

• QUIC-CID-aware sniffer & session keying.
• Cross-instance session handover (shared session state) — still out of scope; the phase 10–13 control plane shares config and health, never sessions.
• eBPF/XDP pre-filter to drop floods before user space.
• Optional TLS/DTLS wrapping (proxy terminates, backend plain).
• wayhouse-ui leftovers — phase 10+11 slice 11f shipped a functional PoC; the visual redesign (commit d86c786, docs/10 “The admin GUI” → Redesign) and then confirmation dialogs for destructive actions, in-flight button states, a vitest frontend suite and a table-driven header-forwarding contract test (crates/wayhouse-ui/src/proxy_util.rs) landed. Still open: a confirmation step on config submit, and browser-level end-to-end tests.
• deploy/ — reference Dockerfile (five targets on distroless), compose control-plane demo and plain k8s manifests, smoke-tested by the CI deploy job (docs/12; blocking since 2026-10-02).
• Phase 14 follow-ups — the multi-proxy AllowedIPs fix and the controller-allocated tunnel address authority landed 2026-10-02 (docs/11 “Address authority”); the CI-friendly tunnel e2e exists (make tunnel-e2e, CI job tunnel). Lease expiry and tunnel-pool pruning landed 2026-10-04; live address change remains (GitHub issue #40); the UI view landed 2026-10-03, releasing from it 2026-10-04. IPv6 tunnel networks and IPv6 underlays landed 2026-10-03 (docs/11 “Address authority”, spec docs/superpowers/specs/2026-10-03-ipv6-tunnel-design.md), with --tunnel-readdress for a changed network.

Milestone cuts

• MVP: phase 0–2 (L4 TCP+UDP, static, health, metrics).
• v1.0: + phase 3–5 (routing, resolver, zero-downtime).
• v1.1: + phase 6–7 (client IP, hardening).
• v1.2: + phase 8 (discovery, HA operations docs).
• v1.3: + phase 9–10+11 (sniffer loader; single-controller + single-aggregator PoC — fleet config distribution & operational Web UI). Both additive — the data-plane contract is unchanged.
• v2.0: + phase 12–13 (fleet hierarchy / HA / shared intent; regional health fabric) — the multi-region, no-single-point-of-failure realization of the phase 10+11 PoC — see 10-distributed-control-plane.md.
• v2.1: + phase 14 (WireGuard backend transport — wayhouse-agent, backend / proxy-peers registries; proxies reach origins on a different network) — see 11-backend-transport.md.
• v0.3 (in progress): web front door — Caddy packaged behind a web pool (issue #297): web: config (#360), generated via: wayhouse loopback listeners, GET /web and the end-to-end tests (#368 to #370) are built; the renderer (wayhouse-web-sync), the Caddy image and ACME are separate slices.

09 – Technology choices

Language

Recommendation: Rust

• Predictable latency with no GC pauses — critical for the added RTT (NFR N1/N2).
• Zero-copy data path (splice, sendmmsg) maps well; fine control over allocations (slab, buffer pools).
• Safe parsers for peek/sniffers (no buffer overflow in the exposed path).
• Ecosystem: tokio / monoio (io_uring), socket2, nix, arc-swap, hashbrown, governor (rate limit), prometheus/metrics, rustls (only if TLS wrapping), tonic (gRPC resolver).

Alternative: Go

• Faster to develop, very good networking standard library, easy cross-compilation.
• GC pauses are small today but measurable under millions of sessions; more tuning needed.
• Fine if the team is Go-oriented and there is headroom on p99 < 2 ms.

Not recommended

• C/C++: maximum control, but a safety risk in the exposed parser path.
• Languages with a heavy runtime (JVM, Node) for the data path — only conceivable for the control plane.

Runtime / IO model

Plan: v1 on tokio with a thread-per-core layout (SO_REUSEPORT, a LocalSet per worker). An io_uring backend as a later optimization behind an IO abstraction.

Key libraries (Rust)

As-built, not the original pre-implementation shortlist — several items below were planned to use an external crate and shipped as a small hand-rolled equivalent instead, once the actual shape of the problem (tiny backend sets, one map per worker, no need for a general-purpose engine) made the extra dependency not worth it. Still-open v2 choices are marked as such.

• Sockets/syscalls: socket2 0.6 (bind options incl. IP_FREEBIND, IP_TRANSPARENT v4/v6 via SockRef), nix (IP_PKTINFO / IP_ORIGDSTADDR recv + reply cmsgs; recvmmsg UDP ingress — ADR 18; sendmmsg UDP egress — ADR 31; splice TCP pump — ADR 17).
• Data structures: plain std::collections::HashMap for the UDP session table (worker-local, no sharding needed — hashbrown was considered but adds nothing at this scale). ACLs are a hand-rolled binary radix trie over address bits (wayhouse_config::CidrSet) rather than the ip_network_table crate — kept the wayhouse-config “serde + thiserror only” dependency rule intact. Consistent hashing is a hand-rolled rendezvous (HRW) hash over the healthy backends (std DefaultHasher) — no hashring dependency; the backend set is tiny, so HRW’s linear scan is cheaper than maintaining a ring. weighted is likewise a plain weighted round-robin over the cumulative-weight line (one atomic tick per selection, no smooth-WRR per-backend state) — blocky ordering is fine at this backend-set size.
• Config: serde + serde_norway, notify for file watch. No env-var overlay exists (the figment idea from the original plan was dropped — the YAML file + SIGHUP/watch reload covers the actual need).
• Snapshot swap: arc-swap.
• Rate limit: a hand-rolled token bucket (wayhouse_core::ratelimit, a Mutex<HashMap<key, Bucket>>, monotonic refill) rather than governor — one bucket type, two key shapes (per-IP / per-net), not worth a general GCRA dependency.
• Metrics: metrics + metrics-exporter-prometheus.
• Tracing: tracing + tracing-subscriber. No opentelemetry exporter — /metrics (Prometheus) is the only telemetry sink today.
• Resolver: reqwest (HTTP) + tonic/prost (gRPC), both in the wayhouse binary only; wayhouse-core defines the Resolver trait. gRPC codegen via tonic-build at build time (needs protoc).
• Admin API: axum (small, on an internal interface).
• Distributed control plane (v2, ch. 10): Tier-1 store — the phase 10+11 PoC’s embedded sled (ADR 20) is kept for HA too, replicated via embedded openraft (ADR 21) rather than switched to etcd or git; Tier-2 health gossip — foca (SWIM) with a hand-rolled (instance, backend) LWW register piggybacked on its broadcast/ anti-entropy (ADR 24, phase 13, fully built). Lives in wayhouse-core itself (wayhouse_core::gossip), not the wayhouse binary — plain UDP, no HTTP/CLI surface, so it fits the crate boundary rule the same way health.rs already does. RBAC password hashing — argon2 (ADR 23), in wayhouse-ui. Phase 12 (hierarchy, both relay logs, adoption, HA, staged/canary rollout, RBAC — ADRs 20–23) and phase 13 (ADR 24) are both fully built — docs/08-roadmap.md.
• Backend transport (phase 14, ch. 11): defguard/wireguard-rs (kernel-netlink primary, boringtun userspace fallback, behind one API) for WireGuard interface management — used by the wayhouse-agent crate and wayhouse’s --tunnel-* reconcile task (ADR 25, built). No new crypto/framing code: the tunnel protocol is unmodified WireGuard.
• PROXY protocol: a small custom v1/v2 encoder (wayhouse_core::proxy_protocol) — no ppp dependency; write-only (the proxy never parses an inbound header) made the hand-rolled version simpler than adopting a parsing-capable crate.
• Tests: a custom Stats/percentile harness (wayhouse-bench) rather than criterion — matches the project’s own latency-ledger reporting shape instead of criterion’s statistical-comparison model; cargo-fuzz for the untrusted-input parsers (crates/wayhouse-config/fuzz/).

Platform

• Primarily Linux (x86-64 + arm64). Kernel ≥ 5.10 for a stable io_uring option.
• Kernel features used: SO_REUSEPORT, splice, recvmmsg/sendmmsg, IP_TRANSPARENT/TPROXY, TCP_NODELAY, SYN cookies, NOTRACK.
• Container: distroless/scratch image, only the required capabilities.
• No Windows/macOS in the data path (only a dev build with a reduced fast path).

Architecture decisions (ADR short form)

Risks & mitigations

10 – Distributed control plane (v2)

Status

Phases 10 through 13 are all now built — see the phase-by-phase status notes below and 08-roadmap.md for the slice-by-slice record. This chapter remains the design source of truth for the mechanisms those phases implement. The v1 data plane and the single-instance control plane (chapters 02, 05, 06) are unchanged by this: every mechanism below is either an additional writer feeding the same validated Snapshot swap, or an additional input to the same per-backend health flag. The hot path gains no network call and no lock.

The controller and aggregator are each a recursive tree of tiers (one tier per failure domain, down to as small as one instance), so the same design covers a single-instance install and a globally distributed fleet with no separate mechanism per size — see “Fleet topology” below. Control (Tier 1) pulls root → leaf with a static standalone/slave role per tier; the aggregator pushes leaf → root with a homogeneous schema at every hop. Locked for the first release (phase 10+11 in docs/08-roadmap.md): one standalone controller tier + one aggregator tier, replicas: 1 each, no slave role, no HA, no adoption, operator intent stays per-instance as it is today (only structural config moves into the controller). The hierarchy, intra-tier HA, adoption, and shared intent are phase 12 — a real, already fully designed extension, deliberately built after something works end-to-end, not before.

Phase 12 status (2026-09-05): fully built, all 8 slices (hierarchy, both relay logs, adoption, intra-tier HA, staged/canary rollout, RBAC + audit) — see docs/08-roadmap.md.

Phase 13 status (2026-09-05): fully built, all 5 slices (config schema, SWIM membership via embedded foca, the per-backend health broadcast, Backend/health.rs integration, and multi-process live verification in crates/wayhouse-fleet-tests) — see “Mechanism (built)” under “Tier 2 — regional health fabric” below, ADR 24 in docs/09, and docs/08-roadmap.md for the slice-by-slice build record.

Why

Today every proxy instance is fully independent (ADR 4):

• Structural config is a YAML file per instance, hot-reloaded on SIGHUP / file-watch. Distribution (image bake, ConfigMap, consul-template, Ansible …) is the operator’s problem.
• Runtime state splits in two, and neither is shared or persisted:
  • operator intent — the phase-5 backend overlay (POST/DELETE a backend), backend admin state (draining / disabled), route hints. Lives on live atomics / ArcSwap tables. Lost on restart. Applied one instance at a time by whoever calls that instance’s admin API.
  • observed health — the active/passive checker’s AtomicBool per backend. Each instance computes it locally.

That is fine for a handful of hand-managed instances. It does not give a fleet operator (or a GUI that “manages everything”):

• a single authoritative place to view and edit config,
• changes that apply fleet-wide and survive restarts,
• an audit trail,
• health that benefits from more than one vantage point.

This chapter plans the smallest set of additions that provide those without touching the data path or the “one immutable snapshot” invariant.

Principles

1. The data plane does not change. The hot path still reads a local immutable Snapshot plus local runtime state. No per-connection call to a store, no added lock. Everything here is control plane.
2. One authority per fact. Every piece of state has exactly one writer tier (below). No fact is writable from two places.
3. Level-triggered everywhere. Sources publish the current value; the instance reconciles by diffing against what it holds. Idempotent, and the same shape as ListenerManager::reconcile and the discovery adapters (phase 8).
4. Degrade to last-known-good. A control-plane outage freezes state, it never clears it — the same rule a flapping discovery source already has to obey (“don’t nuke the pool”).
5. Pull, not push. Instances subscribe / poll. An instance that was down catches up from its cursor on reconnect. No “did the push land” bookkeeping, no per-instance delivery state on the writer.

The two-axis split

Map the new state onto the predicate the data plane already computes:

Backend::takes_new_sessions()  ==  is_healthy()  &&  admin_state() == Enabled
                                   └── observed ──┘   └────── intent ──────┘

Plus a third bucket that stays purely instance-local, never shared: per-listener rate-limit / concurrent-connection buckets (sized per node by design), UDP session tables (worker-local; cross-instance session handover remains out of scope — see ADR 4), per-connection state.

The split is not a new concept bolted on — it follows the seam the code has had since phase 5.

Tier 1 — global config & intent store

Contents

• Structural config: listeners, routes, pools (with their static targets), balancers, filters, resolvers, settings — everything chapter 05 puts in the YAML file.
• Operator intent:
  • backend membership overrides (the phase-5 BackendOverlay),
  • backend admin state (draining / disabled, plus a new fleet-wide force-down),
  • route hints (POST /route-hint today),
  • resolver pins (future — a manual src → pool/target override that outlives a single hint TTL).
• Fleet feature flags.

Shape

• One logical writer — the controller (below). Readers are every proxy instance.
• Ordered, versioned, durable log. Each accepted change is a monotonic revision. Quicksilver-style: central write, asynchronous fan-out, and every instance keeps a local materialized replica that it reads in-process.
• A proxy subscribes on startup: it receives a full snapshot + a revision cursor, then a change stream. On disconnect it reconnects from its cursor; on a total store / controller outage it serves its last replica indefinitely.
• Each received revision goes through the existing validate() → Snapshot::build → ArcSwap::store path. An invalid revision is rejected and logged, the previous snapshot kept — byte-for-byte the behaviour of a bad file reload today. Non-structural intent (admin state, hints) updates the live atomics / tables directly, exactly as the admin API already does.

Relationship to the YAML file

• The file stays a first-class config source. An instance setting config_source: file | store | file+store selects the base; in file+store the file is the structural base and the store is an overlay (matching how the phase-5 overlay already layers on the file).
• The store is authoritative for intent even in file mode, so a restarted instance recovers the overlay / hints / admin-state it had. This closes the “runtime mutations don’t persist” gap directly.

What it is not

• Not on the hot path. Not consulted per connection or per datagram.
• Not a session store. Not shared session state (ADR 4 stands for sessions).

Tier 2 — regional health fabric

Contents

• Per-backend reachability as observed within one failure domain: a small record per (instance, backend) — up | down, last change, check age.
• Later, optionally: per-domain capacity / load signals for smarter load shedding.

Failure domain = reachability-equivalence class

• The set of instances that reach the backends over the same network path — an AZ, a region, a rack row. Set per instance (failure_domain: / region: in settings), or discovered.
• Rationale. Sharing “backend X is down” is only valid between instances whose view of X is the same. Across regions it is noise: a DC1 ↔ X partition says nothing about DC2 ↔ X, and acting on it would wrongly pull X for DC2.

Shape

• AP, not CP. SWIM-style membership + gossiped per-backend state, or a CRDT (a per-(instance, backend) last-writer-wins register). Small, ephemeral, partition-tolerant, self-healing. No election. No quorum write.
• Fully rebuildable. If the fabric is empty (cold start, total partition), each instance simply uses its own checks — precisely today’s behaviour.
• Authenticated. Messages are signed / carried on a mutual-TLS mesh so a rogue host cannot inject “everything is down”.

How an instance decides health (authority model)

Local checks stay in charge; the domain view is quorum-weighted advice:

• Mark unhealthy when this instance sees fall consecutive failures OR a quorum (e.g. > ⅔) of the domain’s instances currently report it down.
• Return to healthy only when this instance sees rise consecutive successes — never on someone else’s say-so (a stale “up” must not revive a backend this instance genuinely cannot reach).
• A Tier-1 force-down always wins over any Tier-2 “up”.

This removes both failure modes: one flapping instance cannot poison the pool (quorum gate on “down”), and one instance cannot ignore a domain-wide outage (the OR).

Mechanism (built, 2026-09-05 — designed and implemented same day)

Membership: embedded foca (SWIM), not hand-rolled. Same reasoning ADR 21 already used for Raft: SWIM’s subtlety — indirect probing, suspicion timeouts, incarnation numbers to un-suspect a member that was never actually down — is exactly the kind of protocol correctness this codebase defers to a maintained, audited crate for rather than reimplementing. foca is transport-agnostic (an application provides the socket and timers), which fits this codebase’s existing preference for owning its own plain UDP socket (net.rs) rather than depending on a batteries-included clustering framework. One foca::Foca instance per wayhouse process when gossip is enabled, identified by (instance_name, gossip bind addr).

Application payload: a per-backend LWW register, piggybacked on foca’s own broadcast/anti-entropy. foca’s BroadcastHandler extension point already re-gossips “active” application data alongside membership traffic with a decreasing invalidation count until it’s fully propagated — exactly the anti-entropy behaviour Tier 2 needs, so no second gossip pass or separate CRDT sync framework is layered on top. The payload itself:

#![allow(unused)]
fn main() {
struct BackendHealthRegister {
    addr: SocketAddr,   // the backend this is about
    up: bool,           // this instance's own local health.rs verdict
    changed_at: u64,    // this instance's monotonic clock at the last flip
    origin: InstanceId, // who is asserting this
}
}

Merge rule: last-writer-wins by changed_at, ties broken by origin — fully deterministic, no coordination, no vector clock. An instance only ever publishes a register for a backend it actually health-checks itself — it never relays or invents an opinion about a backend outside its own snapshot, which is what makes “everyone in a domain reaches the same backends the same way” (the failure-domain definition above) hold in practice, not just in theory.

Wire format and auth. postcard (new workspace dep) for a compact binary encoding of both foca’s envelopes and the broadcast payload — no schema evolution concerns here the way wayhouse-config’s YAML has, since this is ephemeral wire state, not anything persisted. Every datagram carries an HMAC-SHA256 tag (new hmac dep; sha2 is already a workspace dependency via crates/wayhouse/Cargo.toml, promoted to a wayhouse-core dependency too) computed over a per-domain pre-shared key (settings.gossip.psk); a bad or missing tag is dropped silently and bumps a metric, the same “malformed input is discarded, never trusted, never a panic” posture sniff.rs already has for sniffer input. Deliberately not full mTLS: a client-cert mesh is proportionate to Tier 1, which actually accepts writes; Tier 2 is advisory-only and can never independently move traffic (see the authority model above — it can only nudge a decision local checks already gate), so a shared-secret HMAC is the right amount of ceremony for what’s at stake, not the maximum available.

What the shared key does not give is a per-instance identity, so the vote registers are bounded rather than authenticated: a register counts only if its origin is a member the mesh currently knows (this instance included), its changed_at may sit at most a minute ahead of the local clock, and the map holds at most 256 origins per backend and 8192 backends. The origin is not bound to the sender, because members relay each other’s broadcasts. A holder of the key therefore does not even need to join: they can send registers that claim the origin of an existing member, which counts as that member’s vote, and a changed_at up to a minute ahead holds off that member’s real updates for that long. A holder who also joins under several identities can vote with each of them. Tier 2 is advisory (it can only nudge a decision the local checks already gate), so this is accepted for 0.2.0 rather than answered with per-member signatures and new key material. Treat the key like the other fleet secrets and keep it out of instances you do not trust.

Config schema (built exactly as designed — settings.gossip.psk is a plain string, not env-substituted like the ${GOSSIP_PSK} sketch below; this codebase has no env-substitution mechanism anywhere else in wayhouse-config either, so that stayed a design-stage placeholder, see config.example.yaml for the real, commented-out syntax):

settings:
  failure_domain: "eu-west-1a"     # optional; omitted = gossip fully disabled,
                                    # byte-for-byte today's behaviour
  gossip:
    bind: "10.0.1.4:7946"           # this instance's own reachable address (its
                                    # identity and vote origin; 0.0.0.0 is refused)
    seeds: ["10.0.1.5:7946", "10.0.1.6:7946"]
    quorum_fraction: 0.66           # default; the ">⅔" from the authority model
    psk: "${GOSSIP_PSK}"

validate() rejects gossip set without failure_domain or vice versa — a domain identity with no mesh, or a mesh with no domain identity, is always a misconfiguration, never a valid degenerate case (unlike standalone with no children, which is a valid degenerate case for the controller).

Health integration — additive, not a rewrite of pool.rs. Backend gains one new field, domain_down: AtomicBool, alongside the existing healthy: AtomicBool (untouched — still driven only by observe()’s local rise/ fall streaks, exactly as today). is_healthy() becomes healthy.load() && !domain_down.load(). A new wayhouse-core::gossip module (control-plane, sits next to health.rs, spawned by runtime.rs only when settings.gossip is present) owns the foca instance, maintains the merged per-backend register map, computes the domain-quorum verdict per backend on every membership change, and calls a new Backend::observe_domain(quorum_down: bool) that only ever sets domain_down — it can never clear healthy, and can never set healthy true on its own, matching “return to healthy only on this instance’s own rise streak” above word for word. AdminState::Disabled (Tier-1 force-down) already gates selection independently of both flags, so “Tier-1 force-down always wins” needs no new mechanism — it already does, today.

New metrics (metrics_defs.rs, once built): wayhouse_gossip_members (gauge, per domain), wayhouse_gossip_messages_total (counter, sent/received), a per backend wayhouse_backend_domain_down (gauge 0/1 — whether the domain quorum is currently overriding to down), and wayhouse_gossip_auth_rejected_total (counter — bad HMAC or malformed datagram).

Rejected alternatives (full reasoning in ADR 24, docs/09): hand-rolled SWIM (reimplements exactly the subtlety a maintained crate already gets right); etcd/Consul-backed membership (a real extra service for an ephemeral, advisory, AP-by-design signal — contradicts both the “embed, don’t add a service” thread through ADRs 20/21 and Tier 2’s own AP framing above); full mTLS (real certificate lifecycle for a signal that can only nudge, never independently act); a general anti-entropy/CRDT-sync library on top of foca (foca’s own broadcast invalidation already gives eventual convergence for a payload this small — a second framework would duplicate it).

Explicitly out of scope for the first slice of this feature (docs/08 Phase 13, same “don’t half-land a phase” discipline as everywhere else): per-domain capacity/load signals (docs/10 already flagged these as “later, optionally”); failure_domain discovery (cloud-metadata autodetection, etc.) — v1 is configured-only, matching the same “never inferred, always explicit” bias already used for the controller’s standalone/slave role.

Fleet topology: aggregation and control form the same hierarchy, opposite directions

The goal is a single design that scales unmodified from one instance to a globally distributed fleet — not a different mechanism per size. Both new components generalize into a tree of tiers (a tier is typically a failure domain / region, but can be as small as one instance):

• Aggregation (metrics, health, session/pool views) is Push, leaf → root. Each proxy pushes its own state up to its tier’s aggregator; each aggregator, in turn, pushes (or is pulled — see below) a merged view up to its parent aggregator, if it has one. The schema is homogeneous: a parent aggregator’s input (a child proxy, or a child aggregator’s merged view) looks the same either way, so the merge logic is the same function applied recursively — no distinction between “a leaf” and “a subtree” at the parent’s level.

  • Why push, not pull, here (a deliberate exception to principle 5): pull would require every parent to hold a network route down to every leaf, which is the reachability problem the whole hierarchy exists to avoid — a proxy behind restrictive egress-only networking, NAT, or in a partner’s network can push out without ever accepting an inbound admin connection from anywhere, including its own tier’s aggregator. The cost: every proxy/aggregator carries a small outbound client with retry/backoff and a short local buffer so a momentary parent hiccup doesn’t drop a push.
  • Losing the parent link degrades to “my local aggregator still has the full local view; the parent’s view of me goes stale/absent.” No data loss below the cut, no effect on the data plane anywhere.

• Control (structural config + operator intent, Tier 1) is Pull, root → leaf, unchanged from the single-tier design above except generalized to N hops: a proxy subscribes to its tier’s controller (full snapshot + cursor, then a change stream); a slave controller subscribes to its parent the same way and relays what it receives to everything under it. This is principle 5 applied recursively, not a new mechanism.

Controller role: standalone vs. slave — a static, install-time fact

Every controller tier is configured, not elected, into one of two roles:

• standalone — a root authority. It accepts writes directly (config API, intent API) and is the top of its own subtree. A single-instance deployment is a standalone controller with no children and no parent — the degenerate case falls out for free, no special-casing.
• slave — has a parent (parent_addr + a token). It never accepts a write directly; every write it receives from below is forwarded up, and it relays the parent’s revision stream to everything below it. If it loses the parent, it freezes on last-known-good — exactly the existing Tier-1 rule (“a control-plane outage freezes state, it never clears it”), now applying at every hop, not just proxy-to-single-controller.

The role is never inferred from connectivity. A slave losing its parent does not, and must not, promote itself to standalone — that is precisely how two regions would end up independently accepting conflicting intent for the same fact during a network partition rather than a real failure (violates principle 2, “one authority per fact”). Promotion is only ever an explicit operator action (see Adoption, below).

Intra-tier HA is an orthogonal knob

standalone/slave answers “does this tier have a parent.” It says nothing about how many processes back that tier. Separately, every tier picks a replica count:

• 1 node — fine for a single proxy or a small region that accepts freezing (not losing — the last-good state is still served) on that one node’s failure.
• A small consensus group (Raft/etcd-style) — for a tier that must keep accepting new writes (if standalone) or stay live for its children (if slave) through a single node loss. This is the same mechanism either way; a standalone root cluster and a slave regional cluster both run the same consensus group internally, they just differ in whether accepted writes originate locally or get forwarded to a parent.

So a deployment picks two independent settings per tier — role: standalone | slave × replicas: 1..N — never a fixed shape baked into the software. A lone proxy: standalone + 1. A globally distributed fleet: a standalone Raft group at the root, slave groups (sized to taste) at each region.

Adoption (built — phase 12 slice 5)

Status: built (phase 12 slice 5, 2026-09-05) — POST /admin/adopt, see docs/06 “Adoption” and ADR 21/docs/09. Was not part of the phase 10/11 slices. The two “correctness details to solve when this is built” below are both resolved in the shipped version (it refuses with 409 unless the child’s config and intent stores are still empty).

Turning a standalone tier into a slave of a newly-introduced parent — e.g. an operator adding a higher tier’s address + token in the admin UI after the fact, rather than at install time — is a one-time, operator-triggered role flip, not a new mechanism: the child starts the same subscribe flow (full snapshot + cursor + change stream) against the new parent that any slave uses from boot. Two correctness details to solve when this is built, not now:

• The child’s own revision history must not outrank the parent’s once adopted — post-adoption, the parent is the sole authority for every fact the child used to originate itself.
• Adoption should require the child to be quiescent (caught up, nothing in-flight) before the flip, so a write doesn’t get reordered across the transition.

The controller (wayhouse-controller)

A new, optional service, deployed as one tier per the topology above (root standalone, or slave under a parent tier). Proxies never talk to each other through it — it is the Tier-1 writer for its subtree, and (transitively) the GUI’s route to any fact in the fleet.

• Owns its tier’s Tier-1 store — embedded (Raft in the controller), or backed by etcd / a SQL DB / a git repo. This is an ADR-level choice, deferred (see Open questions). A slave tier’s store is a materialized replica of what its parent relays, not an independently-written store.
• Config API (accepted only at a standalone tier, or forwarded up by a slave). Submit a full or partial config; the controller runs the same validate() the proxy does, assigns a revision, publishes down its subtree. History, diff, one-key rollback, staged / canary rollout (a subset of instances take a revision first).
• Intent API. The phase-5 admin verbs (drain backend, add / remove backend, route-hint, …) but fleet-wide (within the tier’s subtree) and persisted — each writes a Tier-1 revision instead of poking one instance. Direct per-instance admin stays as break-glass.
• Auth / RBAC / audit log live here. The proxies’ own admin APIs stay internal-only and are locked to their controller’s identity (or its network); a slave controller’s link to its parent carries its own token, independent of any proxy-facing credential.
• HA is the intra-tier replica-count knob above: N controllers in one tier behind a leader lock for writes originating at that tier; reads are stateless. A whole tier’s outage stops changes for its subtree, not the data plane — every instance under it serves its last replica. See “Intra-tier HA (design)” below for the concrete mechanism (openraft, one Raft group per tier replicating both logs).

The aggregator (wayhouse-aggregator)

A new, optional, stateless service, one instance (or a small horizontally- replicated group — no consensus needed, it holds no durable state) per tier. It is the Tier-2-adjacent read path: fleet view + fan-out for the phase-5 operational verbs, entirely separate from the controller’s Tier-1 write path.

• Ingests pushes from its tier’s proxies and from any child aggregators (a tier’s aggregator is itself a valid “leaf” to its parent aggregator — the homogeneous-schema point above).
• Serves the merged view for its own tier immediately (GET /fleet/*), and separately pushes (or is pulled, if a deployment prefers that at the top where reachability isn’t a constraint) that merged view to its parent, if it has one.
• Fans out the phase-5 intent verbs (drain, add/remove backend, route-hint, drain an instance) to every instance in its subtree at once, reporting partial success per instance rather than failing the whole call.
• Trust model. Two credentials keep telemetry and control apart: --ingest-token unlocks only POST /ingest (what every wayhouse and child tier pushes with), --auth-token gates GET /fleet/* and every fan-out verb; setting --auth-token without a different --ingest-token is refused at startup. A pusher can therefore report state but not drain or edit anyone. Each push’s self-reported admin_url is where the fan-out later sends --instance-token, so it is checked at ingest: with --instance-url-allow (CIDR, hostname or *.suffix, repeatable) its host must match an entry (a hostname entry trusts whoever controls DNS for that name); without, it must be an IP literal equal to the pushing connection’s source address, i.e. an instance can only name itself (a push through NAT, a load balancer or a TLS terminator, or a tier relaying its children, needs the allowlist). A refused push is a 400. The fan-out also refuses to forward a path parameter that is empty, ./.., or holds /, \, ?, # (a 400), and percent-encodes the rest when building the instance URL, so a verb cannot be steered to another admin path.
• Carries no authority — it never decides anything, only observes and relays observations up, and relays operator intent verbs down to the instances that actually hold the atomics. The controller (Tier 1) is the only place a decision is durably recorded.

Intra-tier HA (design)

Status: the wayhouse-controller half is built (phase 12 slice 6, openraft, --ha-node-id/--ha-peers/--ha-token — see docs/06 and ADR 21). The aggregator half below (stateless replicas behind one address) stays design-only — it needs no code, just running N replicas. The heading keeps “(design)” because other docs and code comments cite it by that name.

Answers “how does one tier survive a node loss” — orthogonal to standalone/ slave (that answers “does this tier have a parent”). Two very different components, two very different designs:

The aggregator: stateless replication, no consensus

wayhouse-aggregator already carries no durable state (IngestStore is in-memory, latest-write-wins, deliberately unpersisted — see its own lib.rs). HA for a tier’s aggregator is therefore just running N identical, uncoordinated replicas behind one stable address (a VIP, DNS round-robin, or a k8s Service) — no leader, no consensus, no shared storage:

• Every proxy in the tier pushes to the tier’s one stable address, never to a specific replica — whichever replica the load balancer picks that tick ingests the push. Same rule for a child aggregator’s upward push (--parent-url names the tier’s address, never a node).
• A GET /fleet/* read also goes through the same stable address, so a client sees some replica’s view, not necessarily the freshest — replicas are not required to agree with each other at any instant.
• Accepted inconsistency: two replicas can hold different subsets of a tier’s instances until every instance’s next push interval (default 10s) reaches whichever replica currently answers reads. This is not a regression from the single-replica design — it already showed up as “stale/absent” in the Failure behaviour table, HA just spreads it across replicas instead of across time. Self-heals every push interval with zero coordination code.
• Rejected: gossiping ingested state between aggregator replicas so they converge faster. Adds a real distributed system (membership, anti-entropy) to a component whose entire design point is “carries no authority, forgets nothing important because everything re-arrives on the next tick” — not worth it for shaving one push interval off a view that was never meant to be strongly consistent in the first place.

The controller: Raft, replicated log, single leader accepts writes

wayhouse-controller does hold durable, order-sensitive state (two revision logs), so its HA needs an actual consensus protocol — a tier that must keep accepting new writes (standalone) or stay live for its children (slave) through a node loss needs agreement on “what got written and in what order,” which a stateless replica set cannot give.

• Library: openraft (a Raft implementation, storage-and-transport agnostic), not a hand-rolled consensus algorithm and not an external etcd dependency. Rationale, extending ADR 20’s reasoning: this project has already chosen to embed a KV (sled) rather than run an extra service for the Tier-1 store; embedding Raft continues that “one binary, no external service” story instead of trading it away the moment HA enters the picture. etcd would work but reintroduces exactly the operational dependency sled was chosen to avoid, for a fleet-control-plane component whose own write volume is human-paced (config/intent changes, not a hot path) — a full etcd cluster is a lot of operational machinery for that load. A hand-rolled Raft was never seriously considered: Raft’s subtlety (log matching, leader lease safety, snapshot/log truncation) is exactly the kind of thing worth a maintained, audited crate rather than a bespoke implementation for a security-relevant (per the existing “Tier 1 is a high-value target” security note) component.
• One Raft group per controller tier, replicating both logs together. Not two independent groups (one per log): a tier’s config log and intent log already share a lifecycle (both come from the same set of peers, fail over together, and a slave tier already relays both from the same parent) — one group means one leader election, one peer list, one leader lease to reason about, instead of two that could disagree about who leads. The Raft log’s state machine applies an entry to whichever of the two Stores (config or intent) the entry names; each entry carries a {log: "config" | "intent", bytes} envelope.
• State machine = the existing sled Store, unchanged. openraft’s storage trait needs a state machine and a log store; the state machine is today’s Store::put/get/revisions_after exactly as built (slice 1), fed by openraft applying a committed entry instead of a direct HTTP handler call. The Raft log itself (openraft’s own append-only log of proposals, distinct from the config/intent revision logs it carries) lives in a third sled tree, <data_dir>/raft, on every replica. That log is purged per openraft’s snapshot policy (a snapshot every 5 000 entries, the last 1 000 entries kept); a follower or learner behind the purge point is caught up by snapshot, which replaces its config and intent stores (revision numbers, stage and actor included).
• Writes go through Raft; reads don’t. POST /config, POST /intent, and POST /admin/adopt propose a Raft entry and wait for it to commit before responding (this is where “who is the leader” matters). GET /config, /config/subscribe, /config/revisions*, GET /intent-side reads, and /healthz are served by whichever replica received the request, straight from its own local Store — no ReadIndex / leader-lease read protocol. This is a deliberate relaxation of Raft’s usual linearizable-read guarantee, justified the same way the whole chapter already justifies it: principle 4 (“freeze on last-known-good”) already accepts a subscriber seeing a slightly stale replica during a partition; a follower a few commits behind its leader is the same acceptable staleness, not a new risk. Never a risk of serving state out of order — sled’s local apply only ever appends, so a follower’s view is always a strict prefix of the leader’s, never a divergent one.
• Non-leader write handling: transparent HTTP forward, not a redirect. A replica that receives a write it isn’t the leader for forwards the request to the current leader over HTTP, or HTTPS for https:// peers (openraft tracks the current leader; the follower proxies the request body byte-for-byte, same forwardable_headers pattern wayhouse-ui’s proxies already use) and relays the leader’s response back verbatim. The forward is bounded (10 s, then 504: the write may still have been applied), so a half-open leader never hangs the follower’s handler. Rejected: an HTTP redirect (307 + Location) — every existing client of this API (wayhouse, wayhouse-ui, a human with curl) would need new leader-following logic; a transparent forward means literally nothing downstream of this ADR needs to know HA exists. The one-hop latency cost only applies to writes (rare, human-paced), never to reads or to the data plane. The forward re-sends the caller’s own Authorization, Content-Type and X-Actor; the leader re-checks the token (the --ha-token is never substituted). So every replica must run the same --auth-token, and with plain-http --ha-peers entries the caller’s bearer token crosses the network between nodes in cleartext — use https peers (--ca-file) when that network is not trusted.
• Raft RPCs travel over the same axum server, a separate route prefix (/raft/*), gated by a separate peer-only shared secret (--ha-token, distinct from --auth-token) rather than mTLS or a new transport — mirrors every other cross-service credential in this fleet (--instance-token, --parent-token, …), which are all shared-secret bearer tokens over plain HTTP on a trusted internal network, not a new trust model for this one link. openraft’s async network trait is implemented as a thin reqwest-based client, matching every other outbound HTTP call in these crates.
• A slave tier’s upward relay runs only on the leader (built 2026-10-05). parent_client / intent::relay (slices 1 and 4) become leader-only tasks — every follower would otherwise apply the same parent revision independently and each assign it a different local revision number, corrupting the replicated log. On a leader failover, the newly-elected leader must resume the relay from where the group left off, not where it personally last ran one — so the relay’s cursor (last_applied_parent_revision, one per log) becomes part of the replicated state machine (a small extra key per Store, committed alongside each relayed entry) instead of a process-local variable. This is the one genuinely new piece of state HA introduces beyond “replicate what already exists.” As built: the cursor is a relay_cursor tree beside each log, written in the same sled transaction as the relayed revision and replaced by a snapshot install; the leader proposes RelayConfig/RelayIntent entries carrying the parent revision; the state machine skips an entry whose parent revision is not above the cursor (a deposed leader’s late proposal), so no replica applies a parent revision twice. A new leader seeds a cold tier from the parent’s current config once (cursor 0), then subscribes from the cursor. A --role slave tier keeps the cursor without HA too, so a restart no longer replays the parent’s intent history. POST /admin/adopt is refused on an HA controller: the role is a per-process setting, and a replicated role flip is not built.
• Cluster membership: static at bootstrap, dynamic afterwards (built 2026-10-03). --ha-peers 1=host1:9901,2=host2:9901,... (or 1=https://host1:8443,..., built 2026-10-02 — docs/12 “HA replicas over TLS”) on every replica forms the initial voter set (openraft’s single-step static bootstrap). --ha-peers is only read when a cluster is first bootstrapped (raft.initialize is refused once the log is non-empty), so restarting with a new list changes nothing for existing members. A running group changes through GET/POST /admin/ha/members and DELETE/PUT /admin/ha/members/{id}: a new or replacement node starts with --ha-join (it never initializes) and is added by the leader, which first asks the node who it is (/raft/whoami) so a node id is never pointed at another node’s address; remove a node, or PUT a new address (including an https:// one), the same way. Design: docs/superpowers/specs/2026-10-03-ha-replicated-address-allocation-design.md, recipe in docs/12 “HA membership”.
• The tunnel registries are replicated state too (built 2026-10-03). The backend-peers and proxy-peers registries and the address book are applied by the state machine, next to the config and intent logs, so a registration made on any node is seen on the others with the same tunnel address and a new leader keeps allocating without collisions. The leader records the cluster’s --tunnel-network once (ha/cluster_state.rs); registry entries are applied against that recorded network, never a node’s own flag. On the first HA start the leader imports one node’s pre-HA registry data (ha/import.rs, docs/12 “Upgrading a single controller to HA”).
• replicas: 1 (today’s shape) needs no code path change. A one-node Raft group trivially elects itself leader and commits every entry immediately (no network round trip) — openraft handles the degenerate case for free, so the phase 10+11 single-controller deployment shape isn’t a special case of the HA design, it’s the N=1 instance of it.

What this buys, and what it costs

Per the existing Failure behaviour table, a standalone (or slave) tier’s single-node outage today “freezes changes for its subtree” — every proxy under it keeps running on its last replica, which is already the acceptable degradation this whole design is built around. HA’s benefit is narrower than it might sound: it turns “changes are frozen until an operator restarts the node” into “changes keep flowing through the surviving majority,” for deployments where that freeze window is unacceptable (a large region, or a root tier serving many children). It adds real operational cost (N processes per tier instead of 1, a peer list to manage, a new failure mode — a lost Raft quorum, which degrades to the same frozen-changes behavior a single-node outage already has) for that benefit, which is why docs/08 correctly scopes it as an explicit later slice, opted into per tier via replicas: 2..N, never a default.

Staged / canary rollout (design)

Status: built (phase 12 slice 7) on the controller side — POST /config?stage=canary&group=<name>, GET /config?group=<name>, GET /config/subscribe?since=<rev>&group=<name>, POST /config/promote/{revision}, and the {promoted, canary_groups} stage sled tree. See docs/06 and ADR 22. Not yet wired on the wayhouse side: a real wayhouse --controller instance has no way to enroll itself in a group yet — there is no settings.controller.canary_group config field (the design sketch below), and controller_client does not pass &group=. The feature is exercisable today only with a manual curl subscribe. The heading keeps “(design)” because wayhouse-controller’s module docs cite it by that name.

A subset of a tier’s instances take a config revision before the rest of the fleet — “The controller” section above already named this as part of the Config API; this is the concrete mechanism.

Instances self-report a rollout group

(Design as sketched; the wayhouse-side field below is not built — see the Status note above.) Every wayhouse instance gains an optional settings.controller.canary_group: <string> (default: unset, meaning “not enrolled in any canary group” — the overwhelmingly common case, and the entire mechanism is invisible to an instance that never sets it). controller_client’s GET /config/subscribe?since=<cursor> gains a &group=<canary_group> query parameter (omitted when unset).

Self-reported, not independently verified — the same trust level IngestPayload::admin_url already has in the aggregator: every instance in a tier already holds that tier’s controller/aggregator credentials and is therefore already inside the trust boundary this whole control plane operates within. Verifying group membership against some external source of truth would be a real feature (ties into the open “region-scoped intent” question) but isn’t needed for “let an operator try a change on the three instances they labelled canary before the other three hundred.”

One log, revisions tagged with a rollout stage — not a forked history

A canary revision is still one entry in the same monotonic config log (never a second, parallel history to reconcile later) — it just carries rollout metadata alongside its bytes:

{ "promoted": bool, "canary_groups": ["groupA", "groupB"] }

stored in a new small sled tree (stage, keyed by revision number, alongside revisions/meta) rather than folded into the revision bytes themselves — Store::put’s existing content-agnostic contract (never inspects what it’s storing) stays exactly as it is; the API layer writes an extra stage entry in the same transaction.

• POST /config (today’s plain submission) implicitly writes {promoted: true, canary_groups: []} — byte-for-byte today’s existing behavior when the feature is never used. This is the compatibility anchor: nothing about slices 1–5’s tests or wire shape changes.
• POST /config?stage=canary&group=<name> writes {promoted: false, canary_groups: [<name>]} (repeatable — &group= may appear more than once for a multi-group canary).
• POST /config/promote/{revision} flips that revision’s promoted to true in place — the one exception to “every accepted change is a new revision” in this whole design, deliberately: promotion isn’t new content, it’s a visibility change to content that already exists and already has a revision number: minting a second revision with identical bytes just to mark “this one’s fleet-wide now” would make GET /config/revisions show two entries for one real change, and would need its own reconciliation if a promoted revision needs a further canary step (it doesn’t — promotion is terminal).

What a subscriber actually receives

“Current, for group G” is defined as: the highest-numbered revision that is either promoted or has G in its canary_groups. GET /config?group=<G> and the /config/subscribe?group=<G> catch-up range both compute this by scanning stage from the current pointer backward until a match — accepted as an O(revisions) operation rather than adding an index, matching this codebase’s general bias against building for a scale problem that doesn’t exist yet: this runs at human-paced write rates against a log sized in the hundreds to low thousands of entries for any deployment this design targets, not a hot path and not wayhouse-core. Revisit with an index (e.g. a sled tree keyed by group, pointing at its latest visible revision) if a real fleet’s revision count ever makes the scan measurable.

An instance with no canary_group set behaves exactly as it does today: it only ever sees promoted revisions, so an in-flight canary is completely invisible to the rest of the fleet until someone explicitly promotes it.

Rollback and diff are unaffected; intent is explicitly out of scope

POST /config/rollback/{revision} keeps re-submitting the target revision’s exact bytes as a new, immediately-promoted revision (a rollback is presumably urgent, never something to stage) — no change to slice 5’s rollback semantics. GET /config/revisions gains promoted and canary_groups columns; diff is unaffected (it already compares two revisions’ bytes, unrelated to visibility).

Canary staging applies to config revisions only, not the intent log — a deliberate scope cut, not an oversight. An intent op is already a narrowly targeted, single mutation (add one backend, patch one backend’s state); “give this whole document to a subset of the fleet first” has a clean meaning for a structural config swap and no equally clean one for “apply this one backend-add to a subset” (an add/remove is either safe everywhere it makes sense or it isn’t — there’s no natural partial-rollout story for a single op the way there is for a whole document). Revisit if a real use case surfaces; not designed here.

RBAC and audit (design)

Status: built (phase 12 slice 8) — wayhouse-ui --users-file (argon2 hashes, viewer/operator/admin), --ui-password kept as legacy single-secret mode, X-Actor on every proxied write, and the controller’s per-revision actors sled tree (GET /config/revisions gained an actor field). See docs/06 and ADR 23. The heading keeps “(design)” because other docs cite it by that name.

docs/10’s original text placed “Auth / RBAC / audit log” on the controller. The phase 10+11 slice-11 redesign (see HANDOVER.md) already diverged from that once, for a good reason that still holds: wayhouse-ui is the only place a human ever authenticates — the controller and aggregator each still gate on one shared machine bearer token apiece, and that stays true here too. So this design keeps that split and completes it: RBAC is enforced in wayhouse-ui, the one place with human identity; the controller gains only what it’s missing to make the resulting audit trail meaningful — knowing who (not just that) a revision came from.

Multiple human identities, replacing the single shared password

wayhouse-ui --ui-password (one shared secret for every operator) is replaced by --users-file <path>, a YAML list:

users:
  - username: alice
    password_hash: "$argon2id$v=19$..."
    role: admin
  - username: bob
    password_hash: "$argon2id$v=19$..."
    role: operator

• argon2 (the argon2 crate, pure Rust) for password hashing — the conventional, currently-recommended choice for this exact job; not bcrypt (older, smaller work-factor ceiling) and not a hand-rolled scheme (this is precisely the kind of security-critical, well-solved problem this codebase already defers to a maintained crate for, the same reasoning ADR 16’s sniffer sandboxing and this session’s HA openraft choice both use).
• A companion wayhouse-ui --hash-password CLI mode (prints an argon2 hash for a password read from stdin) is the intended way an operator populates users-file — never handling plaintext passwords in a config file at rest.
• --ui-password is retained as a legacy single-shared-secret mode (mutually exclusive with --users-file) rather than removed outright: unlike this codebase’s usual “clean break over compat shim” pre-1.0 stance, forcing every existing single-operator deployment to mint a users-file for a single-user setup is pure friction with no correctness upside — RBAC only matters once there is more than one identity to distinguish. A single-password deployment implicitly runs as admin.

Three roles, matching the verb tiers the API surface already has

• viewer — every GET: fleet reads, config/revision reads and diffs. No verb that changes anything.
• operator — viewer + the phase-5 intent verbs already fanned out through the aggregator (drain / undrain an instance, backend add / patch / delete, route-hint) and their controller-intent-log equivalents (slice 3). Cannot touch structural config or fleet topology.
• admin — operator + config submit / rollback / promote (staged rollout, above) and POST /admin/adopt (a role/topology change, correctly gated at the top).

Three roles, not a fully general permission-matrix model: every route in wayhouse-ui’s existing surface already falls cleanly into one of these three buckets (see docs/06’s “Fleet control plane” endpoint reference), so a richer model would be solving a problem this API surface doesn’t yet have. Region/subtree-scoped roles (“operator, but only for region X”) are explicitly deferred — wayhouse-ui only ever talks to the root tier (per the existing “GUI served only by the topmost tier” rule) and today’s intent verbs have no region selector to scope against (the same open question docs/10 already tracks as “Region-scoped intent”); scoped RBAC is only meaningful once that lands.

Enforcement point and session shape

session::SessionStore’s HashSet<String> (a session id is either valid or it isn’t) becomes a HashMap<String, Role> (a session id maps to the role its login resolved to). auth::require_session is joined by a new require_role(min: Role) middleware, layered per route group in api::router exactly the way require_session itself is layered today — viewer-level routes get require_session alone, operator-level routes get require_session then require_role(Operator), admin-level routes require_role(Admin). A role check failing is 403, distinct from require_session’s 401, so the frontend can tell “not logged in” from “logged in, not allowed” and render accordingly.

Audit trail: the controller learns who, not just what

Every accepted revision (config or intent) already answers “what changed and when” via GET /config/revisions. It has never recorded who submitted it — wayhouse-controller only ever sees wayhouse-ui’s single shared --controller-token/--aggregator-token, identical for every human behind it. Closing this gap needs one addition at each hop:

• wayhouse-ui sets a new X-Actor: <username> header on every write it proxies to the controller or aggregator (aggregator_proxy/ controller_proxy, using proxy_util’s existing header-forwarding path in reverse — an outbound header added, not an inbound one preserved).
• wayhouse-controller’s submit()/intent submit_intent() read X-Actor ( Option<String>, defaulting to "unknown" for a request that didn’t carry one — e.g. a direct curl against the controller’s own token, which stays valid break-glass access and simply produces a less specific audit entry) and store it in a new actors sled tree, keyed by revision number, alongside stage — same pattern, another small parallel tree rather than widening Store’s core content-agnostic contract.
• GET /config/revisions gains an actor field per entry. wayhouse-aggregator does the same for the fan-out verbs it forwards, logging (instance, actor, verb, timestamp) — in-memory only, matching its existing “carries no durable state” design; a durable fleet-wide audit log (as opposed to “the last actor value attached to a controller revision, which is already durable via sled”) is not designed here — the controller’s revision history already gives a durable, complete audit trail for every fact that has lasting effect (config, intent); the aggregator’s own verb log is a convenience for “who just did this,” not a compliance record.

This is intentionally a thin audit story, not a full authorization/audit framework — proportionate to a three-role model with one human-facing process, not a multi-tenant compliance product.

The admin GUI

Served only by the topmost standalone tier — never by a slave tier. Every regional controller/aggregator stays a machine-to-machine API (serving its own subtree, relaying to its parent) with no browser-facing frontend at all; only the root hosts the GUI, because it’s the one place with full-fleet authority and it keeps auth/RBAC/session state a single fact in a single place, instead of N regional logins each needing to agree on who can do what. Serving a UI per tier would reopen exactly the problem the hierarchy was built to close: an operator not knowing which region’s login is authoritative for a given action, and RBAC that must be kept consistent across every tier instead of living in one.

A dedicated wayhouse-ui process is the GUI’s home — not the controller, not the aggregator. The GUI needs both: reads/operational verbs from the aggregator, config/revisions from the controller. Bolting human login onto either one taxes it with a concern that isn’t its own: a session store is a form of authority (“who’s allowed to act”), which sits wrong on the aggregator (deliberately “carries no authority”); and putting the browser’s whole surface on the controller couples presentation to Tier-1’s write path for no reason. wayhouse-ui is a pure BFF — human session auth in front, holding the controller’s and aggregator’s own machine credentials to call each on the operator’s behalf, authority over neither. It’s the one piece of this picture with no “real” state of its own (no store, no fleet data, nothing that outlives a restart beyond active sessions), so it costs nothing to add as a fourth (later: a third-per-tier) always-optional process; it never gains its own write path around the controller.

A regional tier’s raw API is still reachable directly (curl / CLI, on its own internal network) as break-glass during a root outage or partition — the same status the phase-5 direct-per-instance admin API already has next to the aggregator/controller. That is deliberately not a served web UI: break-glass access is for an operator who already knows exactly what they’re doing to one region, not a second, parallel product surface to build and secure.

Two capability levels, shippable in order:

1. Operational — needs no Tier-1 store. Fleet-wide view + the phase-5 verbs, via the stateless aggregator hierarchy. This is roadmap Phase 10 and is the “Web UI for the admin API” listed in chapter 08’s Later / optional. A first deployment is a single aggregator tier — the hierarchy above is what a deployment grows into with more regions, not a prerequisite for phase 10’s first release.
2. Full management — structural config editing, revision history, rollback, diff, RBAC. Needs Tier 1 + the controller hierarchy. Roadmap Phase 11.

No proxy admin port is ever exposed to a human; the GUI carries all authentication and authorization.

Redesign (2026-09-05): the GUI gained a real fleet tree (instances group under a self-reported settings.group path, e.g. "eu/frankfurt/cluster-a" — a display-only label alongside the existing admin_url/sessions fields in IngestPayload, unrelated to the Tier-1/Tier-2 hierarchy above), a schema-driven settings form over the same submitted YAML text (a raw-YAML panel stays underneath so every field — including ones the form has no control for yet — is always reachable), and a Sniffers page for managing sniffer modules. (Later, 2026-10-03: a read-only Tunnel addresses page over the controller’s GET /tunnel/addresses, docs/11 “Address authority”; and, 2026-10-21, a read-only Certificates page over the controller’s /acme/sites, /acme/gateways and /acme/account, acme.md.) Sniffer management needed new backend surface: wayhouse gained GET/POST /admin/sniffers + DELETE /admin/sniffers/{name} (writes into settings.sniffers.dir, then reuses the existing live rescan-on-reload mechanism — no new hot-reload path), fanned out fleet-wide via wayhouse-aggregator’s POST/DELETE /fleet/sniffers[/{name}] and a targeted GET /fleet/instances/{instance}/sniffers. Turning settings.sniffers on from nothing is still startup-only, per its existing design.

Versioning

Every surface that two components share carries a version, and a node that cannot understand its peer says so instead of misbehaving. The design (the N / N-1 window, the upgrade order) is in docs/superpowers/specs/2026-10-05-component-versioning-design.md; this is what is built.

The controller’s /config* operator routes (except /config/subscribe), the UI, /healthz, /metrics and the aggregator’s /fleet/* routes are for operators and the UI and are never gated.

Rule: a breaking wire change (a removed or retyped field, changed semantics, a new required field, a changed raft RPC payload) bumps PROTOCOL_MAJOR (crates/wayhouse-http/src/protocol.rs), even while the product is 0.x; an additive one (new optional field or route) bumps PROTOCOL_MINOR. Protocol numbers are independent of the product version. The gossip frame is versioned on its own: a breaking gossip change bumps GOSSIP_VERSION (crates/wayhouse-core/src/gossip.rs) and not PROTOCOL_MAJOR, since it does not touch any HTTP route; an additive one needs no bump if older nodes ignore it. A datagram with another gossip version is dropped, so there is no N / N-1 window for gossip: during a rolling upgrade nodes of different gossip versions stop sharing health and fall back to their own checks (the data plane is unaffected). A refused request counts into wayhouse_protocol_mismatch_total{route_group="controller"|"aggregator"|"raft"|"proxy"}.

Peers remember the protocol version each other answered with and gate newer optional fields on it (wayhouse_http::peers); the fleet view flags version skew. The operator runbook, the compatibility window and a table of versions per release are in upgrading.md.

Not covered: the raft log and snapshot storage beyond the store marker, and the WireGuard tunnel itself.

Failure behaviour

Security notes

• Tier 1 is a high-value target — a bad revision can blackhole the fleet. Mitigations: validate() on both controller and instance; a revision must be signed by the controller; an instance rejects a revision that fails a sanity bound (e.g. “a pool that had > 0 targets now has 0” → warn and keep the previous snapshot); staged / canary rollout; one-key rollback to any prior revision.
• Tier 2 is lower value (advisory, rebuildable) but still authenticated — signed messages / mTLS mesh.
• The GUI is the only operator-facing component, and it is served solely by the root tier. All authn / authz lives there, once, rather than being kept consistent across every regional tier’s own frontend.

Open questions

Resolved by this session’s design pass (each now has its own section above, plus an ADR in docs/09): Tier-1 backing store / HA mechanism (sled + embedded openraft, one group per tier, ADR 21) — the “git-as-source-of- truth” option was dropped once the concrete design confirmed sled already does everything needed and a git-backed store would need its own Raft-equivalent replication story anyway; one revision stream or two (kept two, config and intent, confirmed by their independent slice 1/3/4 implementations and reaffirmed by “canary staging applies to config only” in the rollout design — the two logs have different visibility and staging needs, splitting was the right call); adoption flow (built, slice 5).

Still open:

• Tier-2 transport: an existing gossip crate (foca for SWIM, a memberlist-style mesh) vs. a minimal hand-rolled CRDT sync. Measure message volume with realistic backend / instance counts before committing.
• Affinity as Tier-2 state: should the resolver result cache become shared-within-a-domain state (so a rehashed client keeps its instance’s affinity)? Overlaps chapter 03 scheme C. (sticky_key was removed, #54.)
• Region-scoped intent: is “drain backend X” ever meant to apply to one domain only, needing intent records with a failure-domain selector? Also now a prerequisite for region-scoped RBAC (see “RBAC and audit” above, which explicitly deferred it for the same reason).
• Controller ↔ discovery: when phase-8 discovery already supplies backend membership from k8s / Consul / DNS, the controller’s intent surface shrinks to overrides on top of discovery. Confirm the precedence order (discovery ∪ overlay − removed, then admin state) and where it is evaluated.
• Aggregator push transport: a bespoke small protocol (HTTP + a batched JSON body, simplest, consistent with the rest of the admin API) vs. an existing wire format (OTLP for metrics-shaped data) — the OTLP path buys interop with existing collectors but the fleet/pool/session view isn’t metrics-shaped, so a bespoke shape is likely still needed alongside it.
• Push buffering bound: how much history a proxy/aggregator holds locally when its parent link is down before it starts dropping — a fixed ring buffer sized in the same spirit as the existing recv-buffer caps (docs/06), not unbounded growth.
• Dynamic Raft membership was deferred here and is built (2026-10-03): see “Intra-tier HA (design)” above and docs/12 “HA membership”.
• A durable, fleet-wide audit log (as opposed to the per-revision actor field this session’s RBAC design adds, which is durable but lives one field per revision, not as its own queryable log): deferred in “RBAC and audit” above as disproportionate to a three-role, single-human-facing- process model.

11 – Backend transport

Status: built (2026-09-06). All 7 Phase 14 slices are implemented and verified live end-to-end in Docker (see docs/08-roadmap.md “Phase 14” for the slice-by-slice record and HANDOVER.md for the bugs the live run surfaced). This chapter remains the design source of truth for the mechanism. The design was locked in a 2026-09-05 session at the depth of the phase-12/13 pre-build sessions; the “Locked decisions” and “Open questions” sections below are kept as written then.

Problem

Every requirement and architecture decision so far assumes what docs/01-requirements.md states outright under Assumptions: “Backends are reachable over a trusted internal network.” That’s true for a single-region deployment (proxy and game servers in the same DC/VPC). It stops being true the moment the fleet is what phase 10–13 were actually built for: proxy instances distributed globally (edge PoPs, one per region) in front of game servers that are not on those proxies’ local network — a different DC, a different cloud, a rented box at a small hosting provider, sometimes behind a home NAT.

Today connect_backend (crates/wayhouse-core/src/proxy.rs:138) and connect_upstream (crates/wayhouse-core/src/listener_udp.rs:844) do a plain TcpStream::connect/UdpSocket::connect to a bare SocketAddr — there is no seam between “route resolved to this address” and “dial it” for anything else to hook into. health.rs’s active checks dial the same way independently. Closing this gap is Phase 14.

Don’t reinvent the wheel

Before designing anything new, what does this problem already have solved prior art for?

• Steam Datagram Relay (Valve) is the closest large-scale precedent: game clients never see the origin server’s IP; traffic is relayed through Valve’s own network, and origin servers register only with the relay fabric. Proprietary infrastructure, not reusable, but it validates the shape of the answer: a relay/tunnel network in front of an origin that never needs a public inbound port of its own.
• rathole / frp: exactly this problem in miniature (a lightweight, high-performance reverse-tunnel proxy for NAT traversal, written in Rust — rathole specifically). Real, working, maintained tools — but they’re standalone binaries with their own framing/auth protocol, not something to embed as a library, and they know nothing about pools, health checks, or this project’s control plane. Good validation that “origin dials out to a relay with a public IP” is the standard shape; not something to depend on directly.
• WireGuard itself, unmodified, is the actual answer for the tunnel data plane. It already solves the hard, security-critical part (an authenticated, encrypted point-to-point link) as a small, audited, extremely widely deployed protocol — exactly the “defer to a maintained, audited implementation for a subtle, security-relevant mechanism” reasoning ADR 21 (Raft) and ADR 24 (SWIM gossip) already established twice in this codebase. Critically, WireGuard already works when only one side has a public, reachable endpoint: the origin-side peer needs no public IP or open inbound port at all — it just needs to be able to send outbound UDP to the proxy’s known public endpoint once, and WireGuard’s own roaming (the far side updates its record of “where that peer’s packets are coming from” on every received, authenticated packet) plus a keepalive keeps a NAT mapping open indefinitely. This is true even for an origin behind a home router — no port forwarding, no static IP, ever required on the origin side. (The one case this doesn’t cover — both sides behind restrictive/CGNAT-style NAT with no cooperating public peer at all — doesn’t apply here, since the proxy side is edge infrastructure and always has a real public IP.)
• Headscale (a BSD-3 self-hosted Tailscale-protocol coordination server) is the closest prior art for automated WireGuard key/peer distribution at fleet scale — proof that “a control plane that hands out WireGuard peer configs” is a well-trodden, sensibly-scoped idea. Adopting it wholesale would mean either running real Tailscale clients (a large, general-purpose product with its own protocol, unrelated to pools/health/ routing) or reimplementing enough of its coordination protocol to be compatible — neither is smaller than extending the control plane this project already runs (wayhouse-controller) with one narrow registry. Its idea is reused; the software is not.

Conclusion: nothing about the tunnel itself needs inventing. Real, unmodified WireGuard (kernel module where available, boringtun’s userspace implementation as a portable fallback) is the entire data plane. The only genuinely new code this project needs is (a) a small origin-side helper that manages a local WireGuard interface and registers itself, and (b) a narrow extension to the existing Tier-1 control plane (wayhouse-controller) that distributes peer configuration automatically instead of by hand — both thin, both specific to fitting this into pools/health/ routing, neither a reimplementation of cryptography or tunnel framing.

Shape

┌─────────────┐  WireGuard  ┌──────────────┐        ┌──────────────┐
│  wayhouse (edge, │◄───────────►│  wayhouse-agent   │◄──────►│  game server │
│  public IP) │   tunnel    │  (origin,    │ plain   │  process     │
│             │             │  no public   │ local   │              │
│  pool target│             │  IP needed)  │ traffic │              │
│  = tunnel-  │             └──────────────┘        └──────────────┘
│  internal IP│
└─────────────┘
        ▲
        │ peer config (pubkeys, allowed IPs)
        ▼
┌───────────────────────┐
│  wayhouse-controller        │  (existing Tier-1 store, ADR 20/21) —
│  + new "backend peers" │  new small registry, distributed the
│  registry              │  same way config revisions already are
└───────────────────────┘

• The data plane never changes. Once a WireGuard interface for a given origin exists on the proxy host, that origin’s backends are just IP addresses on that interface’s subnet — connect_backend/ connect_upstream/health.rs’s dials need zero code changes, because from wayhouse’s point of view a tunneled backend looks exactly like any other routable SocketAddr. This is the same reasoning that already lets this project’s transparent-proxy mode (ADR 12) and every other listener option layer onto the existing pump without touching it.
• wayhouse-agent (new, small binary, its own workspace crate — locked, matching the wayhouse-controller/wayhouse-aggregator/wayhouse-ui precedent of one small process per concern, keeping WireGuard dependencies out of the proxy-critical wayhouse binary entirely). Runs next to the actual game server process. Creates and maintains one shared local WireGuard interface with every proxy PoP it’s paired with as a peer on that same interface (locked — the standard WireGuard shape: one interface, many peers, each scoped by its own AllowedIPs; per-origin interfaces would multiply routing-table entries for no isolation benefit AllowedIPs doesn’t already give per peer). Registers its public key + which backend address(es) it fronts with wayhouse-controller. It does not proxy or inspect game traffic itself — the WireGuard interface does that, and the real game server binds normally on its own loopback/interface address, exactly as it would with zero knowledge that a proxy is involved.
• Interface management via defguard/wireguard-rs (locked) — a multi-platform Rust library unifying kernel-netlink and boringtun- userspace configuration behind one API, used by both wayhouse-agent (the origin’s interface) and wayhouse’s new reconcile task (the proxy’s shared interface, all origins as peers). One abstraction instead of hand-rolling the kernel-vs-userspace branch ourselves on top of two separately-focused crates (wireguard-uapi + boringtun) — continues this project’s own “prefer a maintained wrapper over hand-rolled plumbing” bias (nix for recvmmsg/TPROXY/splice, ADR 10/12).
• wayhouse-controller gains a “backend peers” registry — a new resource alongside the existing config-revision log and phase-12 intent log, not a replacement for either. wayhouse-agent writes its own registration (pubkey, allowed backend addresses, last-known endpoint); every edge wayhouse instance subscribes to the resulting peer table the same way it already subscribes to config (controller_client.rs’s existing shape, ADR 13) and reconciles its shared WireGuard interface’s peer list to match (new wayhouse task, same shape as controller_client::run, using wireguard-rs above). Revoking/ rotating an origin’s key is a controller-side removal that propagates the same way a config change does — no manual per-proxy key exchange, ever, for a fleet of N proxies × M origins.
• …and a mirror-image “proxy peers” registry, built in slice 7. The bullet above only solves proxy→origin discovery. wayhouse-agent’s “every proxy PoP it’s paired with as a peer” premise two bullets up needs the other direction too: an origin has to learn about every proxy, including ones added after it was deployed, without being restarted. So every wayhouse --tunnel-* instance also registers itself (pubkey + its own public endpoint — always known, unlike an origin’s) with a second registry, and every wayhouse-agent subscribes to it and reconciles proxies onto its own interface, the exact mirror of the first bullet’s mechanism run in the other direction. wayhouse-agent --peer-pubkey/--peer-endpoint (a manual static pin, what slice 6’s first end-to-end verification used before this existed) still work alongside it for a bootstrap proxy or a deployment too small to bother with the registry — they converge to the same interface state a registered proxy would reach anyway, so there’s no conflict between the two paths.
• Edge restarts: a per-process boot_id (built 2026-10-03). A restarted wayhouse --tunnel-* has a fresh interface but no endpoint for any origin (origins may sit behind NAT), so only the origin can re-handshake — and with kernel WireGuard the origin’s session to the old process still looks valid, so it used to wait for the 120 s rekey (~2.5 min outage). Every proxy registration therefore carries a random boot_id (128-bit hex, new per process start; the controller accepts 1–64 of [A-Za-z0-9-] and stores it as-is). wayhouse-agent compares whole registrations, so a changed boot_id re-sets the peer (remove + add), which drops the dead session; the re-added peer’s persistent keepalive starts a new handshake at once. Optional on the wire in both directions: an older proxy sends none (no change from before), an older controller drops it, an older agent ignores it.
• Key material stays out of the config-revision log. A peer table is security-sensitive, high-churn (origins come and go), and has nothing to do with routing/pool structure — it belongs in its own resource on the same Tier-1 store, mirroring how the intent log (phase 12) already lives alongside the config log rather than inside it.
• Config schema: an origin is a new BackendSource (locked) — reuses the existing level-triggered discovery seam (ADR 12a: fetch() -> Vec <SocketAddr>, the runtime diffs it, same reconcile path DNS-SRV/Consul/ Kubernetes sources already use) instead of inventing a parallel “origins: / pools[].origin” concept. A pool referencing an origin looks exactly like a pool referencing any other dynamic source today: pools: [{ name: p, source: my-origin }], backend_sources: [{ name: my-origin, type: tunnel, ... }]. The “tunnel” source’s fetch() returns the origin’s currently-registered, tunnel-internal backend address(es) from the backend-peers registry — no new schema concept, one new BackendSource implementation and one new backend_sources[].type variant. This also means an origin’s backend address(es) can change (the agent re-registers) without touching wayhouse-config at all, the same “discovered set, not a static file list” property every other dynamic source already has.

Address authority (built 2026-10-02)

wayhouse-controller allocates tunnel-internal addresses, so no operator chooses (or mis-chooses) one. Design and decisions: docs/superpowers/specs/2026-10-02-tunnel-address-authority-design.md; IPv6 (built 2026-10-03): docs/superpowers/specs/2026-10-03-ipv6-tunnel-design.md.

• Allocation. Start the controller with --tunnel-network fd49:89c1:4b5e:60::/64 (IPv6, /64 to /120; the default in the docs and examples, and a ULA cannot clash with an origin’s own LAN) or --tunnel-network 10.60.0.0/16 (IPv4, /16 to /30). One family per controller and one address per peer; choose IPv4 when a game server binds 0.0.0.0 only, since it cannot be reached on an IPv6 tunnel address. Either family holds at most 65 534 entries (pins included), because allocation scans the pool; capacity in GET /tunnel/addresses is the smaller of that and the host count. The network and all-ones addresses are never handed out, and IPv4 written as IPv6 (::ffff:a.b.c.d) is refused. A registration that omits its address is allocated the lowest free host address; the allocation is sticky per (role, name). A registration may instead pin an address, which is granted only if free (409 otherwise). One global address space is shared by origins and proxies. Without --tunnel-network the controller is pin-only: it enforces uniqueness but allocates nothing, and accepts pins of either family (mixing them is the operator’s responsibility: peers of different families cannot reach each other).
• Names and keys. A registration name is 1 to 64 characters of A-Za-z0-9._- (subscribe is reserved), so it can always be fetched and deleted by name. One WireGuard public key belongs to one name within a registry: a second name presenting a key that another name holds is refused with 409, because peers are keyed by public key and the two registrations would overwrite each other’s routes. Re-register under the same name to move to a new key; deleting a name frees its key.
• Underlay. Independent of the tunnel’s family: WireGuard endpoints (--endpoint, --tunnel-endpoint, --peer-endpoint, written [2001:db8::7]:51820 for IPv6) and the controller URL (http://[fd99::1]:7070) may be IPv4 or IPv6.
• Changing the network. At startup the controller refuses to run when stored addresses fall outside --tunnel-network, naming up to ten of them. Restore the previous network, or pass --tunnel-readdress: the controller then starts, warns once, and gives each such peer a new address at its next registration. The running peer logs the change and keeps its old address until it restarts, so its traffic is interrupted until then. Peers that are offline keep their old entry until they register again; DELETE the ones that will not come back. --tunnel-readdress is refused without a network and together with --ha-peers or --ha-join, and is harmless when nothing is outside the network.
• Startup. wayhouse-agent and wayhouse --tunnel-* register before bringing their interface up (the answer is its address), persist the answer next to their key, and can start from it while the controller is down. For wayhouse --tunnel-* that fallback is not instant: with the controller down it first spends its ~30 s registration budget, then falls back to the saved address (<tunnel-key-file>.address), logging the last error and, if a pinned address differs from the saved one, that the pin needs a restart once the controller is back. 408/429 count as “controller unavailable”, any other 4xx refuses startup.
• Address changes. Every re-registration (default every 30 s) answers with the address the controller currently holds. If it differs from the one the interface carries (an operator released it, or a lease expired and it was reallocated), the agent or wayhouse moves the interface without a restart, keeping its key, listen port and peers, on both backends: it deletes the old address with a netlink call of its own and assigns the new one (live_interface.rs and netlink_addr.rs in both binaries). The library can add an address but not remove one, and its remove_interface cannot be relied on (it fails in DNS cleanup where no resolver tool exists). The old address goes first, because deleting an IPv4 primary address would otherwise take the new one with it. Established sessions survive. The new address is saved for a restart only once it is up; if assigning it fails the old one is put back and the next re-registration retries. The other side follows on its own: each proxy and origin re-reads the registry and re-routes the /32 (/128).
• Routing. Every peer is a host route (/32, or /128 for IPv6): origins route each proxy’s tunnel address (this fixed the earlier AllowedIPs 0.0.0.0/0 bug where a second proxy stole the first one’s route), proxies route each origin’s. Backends must be on the registrant’s own address; --backends :25565 means “my address, port 25565” (stored as [fd49:89c1:4b5e:60::1]:25565 on an IPv6 network).
• Release. DELETE /peers/{name} / DELETE /proxy-peers/{name} free the address and emit a tombstone that subscribers turn into a WireGuard peer removal. GET /tunnel/addresses lists the table with a stale flag (--tunnel-stale-after, default 14 days) and a daily log warning.
• Lease expiry. With --tunnel-lease-ttl (at least 2h; default 0 = off) an owner not re-registered for that long is released exactly like a DELETE: tombstone, address freed, and a pool’s tunnel source clears the origin’s backends (wayhouse_discovery_refresh_total{result="withdrawn"}). The controller sweeps every tenth of the TTL (between 1 min and 1 h), starting an hour after it starts (or after a node becomes HA leader), so a controller that was down longer than the TTL does not expire everyone before they re-register. Under HA only the leader sweeps and each expiry is a replicated Expire entry that re-checks last_seen when it applies, so a re-registration that commits first wins. An owner that comes back after expiry is allocated afresh and may get a different address. Set the TTL well above the agents’ and proxies’ register interval (the 2 h minimum is only a floor for the HA last_seen refresh); hours to days is typical. A claim is kept even when the same registration then fails (backends not on the claimed address, or a storage error), so the corrected retry gets the same address; a stream of distinct names with bad backends can therefore use up the pool. The registration endpoints are bearer-gated, and DELETE plus the stale flag are the remedy. wayhouse-ui shows the same table on its Tunnel addresses page (GET /api/tunnel/addresses, proxied with --controller-url/--controller-token). Each row has a Release button (admin role, behind a confirmation) that proxies the registry DELETE (DELETE /api/tunnel/origins/{name} → /peers/{name}, /api/tunnel/proxies/{name} → /proxy-peers/{name}).
• High availability (built 2026-10-03). --tunnel-network works with --ha-peers: both registries and the address book are replicated through Raft, so every node serves the same registrations and addresses, and a registration or DELETE made on any node (a follower forwards it to the leader) is allocated once, cluster-wide. The leader records its --tunnel-network when the cluster first initializes and that recorded network applies from then on. Give every node the same --tunnel-network: a node whose flag differs logs an ERROR naming both networks and answers registry writes 503, though an unchanged re-registration is still answered 200. Until the network is recorded (the first seconds of a cluster, or while an upgrade waits for a node, docs/12) registry writes answer 503 (“cluster is initializing its registries”). An unchanged re-registration proposes nothing; an hourly last_seen refresh is the only periodic write. Pin-only mode works the same way. Upgrade and membership notes: docs/12.
• Limits. Transparent mode (transparent: true) does not apply to tunnel backends: config load rejects it on a listener whose pool: or route pool actions use a tunnel source. A resolver route can still return a tunnel pool at runtime; the proxy then connects without the client’s source address, and a client/backend family mismatch on any pool is logged at most once a minute. Dual-stack tunnels and live address changes are future work (GitHub issue #40).

Open questions

• What happens when hole-punching genuinely fails (both the origin’s and its upstream NAT are restrictive enough that even the proxy-has-a-public-IP case doesn’t establish a path — rare, but real for some consumer ISPs/CGNAT)? Out of scope for v1; the honest fallback is a documented limitation, with a relay-of-last-resort (closer to Steam Datagram Relay’s actual shape) as a possible v2, not designed now.
• Tunnel-internal address collision/exhaustion — resolved 2026-10-02: see “Address authority” above.
• Does this replace or sit alongside the existing “trusted internal network” assumption? Alongside — a same-network deployment still needs none of this and keeps working exactly as today; backend transport is additive, opt-in per origin.

Locked decisions (2026-09-05, follow-up session)

• WireGuard interface config: defguard/wireguard-rs, not a hand-rolled wireguard-uapi + boringtun split.
• Interface topology: one shared interface with many peers, not one interface per origin.
• Config schema: an origin is a BackendSource (backend_sources[].type: tunnel), not a new origins:/pools[].origin concept.
• wayhouse-agent: its own workspace crate, not a wayhouse --agent mode.

12 – Container images & deployment

How to package the five binaries for Docker/Kubernetes, and — the part that isn’t obvious going in — how a proxy that needs to listen on many, changing ports actually works once it’s namespaced by a container runtime.

Container images

The reference build is deploy/Dockerfile: one file, a shared rust:1-trixie builder and seven runtime targets (the five binaries, wayhouse-minimal, below, and wayhouse-caddy, web proxy) on gcr.io/distroless/cc-debian13:nonroot (Caddy on its own pinned base; see deploy/README.md; make deploy-images builds all seven; make deploy-scan runs an informational Trivy scan of them and of the lockfiles they’re built from — CI’s deploy and trivy jobs). wayhouse-minimal is wayhouse built with --no-default-features (issue #62): no WASM sniffer loader, gRPC resolver, dns_srv source, Tier-2 gossip fabric or WireGuard tunnel client, so the binary is much smaller. It takes the same arguments as wayhouse; a config or flag that needs a dropped feature (settings.sniffers, a grpc resolver, a dns_srv source, settings.gossip, a tunnel source, --tunnel-iface) is refused at startup and under --check, naming the cargo feature.

The published ghcr.io images are public and need no login to pull.

None of these five binaries shell out to an external command at runtime (WireGuard interface management in wayhouse/wayhouse-agent goes through kernel netlink directly via defguard/wireguard-rs, not the ip/wg CLIs) — so the runtime image needs nothing but the binary and its dynamic library dependencies (glibc). It does not need a system CA bundle either: the HTTP clients (--controller, --aggregator, --tunnel-controller-url, the HTTP resolvers) verify against a root bundle compiled into the binary, plus whatever --ca-file adds — see TLS for the fleet services for what that implies. The builder and runtime base must be on the same Debian release, or a binary can fail to start on an older glibc.

[profile.release] sets strip = true, which trims roughly 15-20% off every binary.

Measured image sizes (docker images, stripped release binaries, deploy/Dockerfile on distroless/cc-debian13:nonroot, CI run 36922142697, 2026-10-01):

For comparison, an earlier 2026-09-07 measurement on cc-debian12 gave 45-65 MB.

The base image dominates, not the binary — debian:trixie-slim alone is ~118 MB before anything is added; distroless/cc-debian13 (glibc + libgcc/libstdc++, no shell, no package manager) is a fraction of that and was confirmed to actually run these binaries (no runtime shell-out, so distroless’s missing shell is never a problem). Trade-off: no docker exec ... sh for interactive debugging, and the nonroot variant runs as a fixed non-root uid (fine here — nothing in this project needs root once its sockets are bound; see --tunnel-*’s capability requirements below for the one exception).

wayhouse and wayhouse-controller are the two heaviest either way, for real reasons: wayhouse links wasmtime (phase 9 sniffers) and defguard_wireguard_rs/boringtun (phase 14 backend transport); wayhouse-controller links openraft + sled for the HA/Raft log store (phase 12). wayhouse-agent/wayhouse-aggregator/wayhouse-ui don’t carry those.

Examples

deploy/compose/ is a runnable control-plane demo (controller, aggregator, UI, one wayhouse pulling its config from the controller) with a tunnel override; deploy/k8s/ has plain manifests (the proxy as a hostNetwork DaemonSet, plus the RBAC its kubernetes sources need). Both are reference only (the images themselves are published to GHCR on version tags by .github/workflows/release.yml). CI smoke-tests the compose demo and schema-validates the manifests (the deploy job, blocking since 2026-10-02; first green run 2026-10-01). Aggregator intent fan-out (drain etc.) goes to the admin_url each wayhouse reports, by default http(s)://<settings.admin.listen>; when that address is not reachable from the aggregator (0.0.0.0, loopback, a container port mapping, a TLS terminator in front), set wayhouse --aggregator-admin-url. The k8s DaemonSet reports http://<node IP>:9900, and the aggregator presents --instance-token (the admin auth_token). The compose demo cannot use fan-out: its wayhouse admin API listens on the host’s loopback only, which the bridge-networked aggregator cannot reach.

The aggregator separates the credentials: --ingest-token is what the wayhouse instances push with (--aggregator-token on wayhouse; it unlocks only POST /ingest), --auth-token is for wayhouse-ui and operators (/fleet/*). Setting --auth-token requires a different --ingest-token; the aggregator refuses to start otherwise. It only stores an admin_url whose host is the pushing connection’s own source IP, or one covered by --instance-url-allow (CIDR, hostname or *.suffix, repeatable); set the allowlist when the aggregator sees a different source address than the one reported (NAT, a load balancer, a TLS terminator, the compose bridge, or a child tier relaying through --parent-url). A refused push gets a 400 and a warning in the aggregator’s log. A hostname entry trusts whoever controls DNS for that name, so prefer CIDRs where the addresses are stable. See docs/10, “The aggregator”.

Networking: a proxy that binds many, changing ports

wayhouse is designed to add and remove listeners live — ListenerManager (crates/wayhouse-core/src/listeners.rs) reconciles the running listener set from config on every SIGHUP/reload, and bind: "0.0.0.0:30000-30999" (docs/01 F1.4, docs/08 roadmap) spawns one real socket per port in a range under a single listener config, for fleets that hand out one port per match/instance. None of that is Docker- or Kubernetes-aware — it’s just bind()/listen() calls inside the container’s network namespace. And that’s exactly the problem: neither platform lets a running container dynamically “claim” a new externally-reachable port just by calling bind(). Port exposure is a static declaration in both:

• Docker: -p hostport:containerport sets up a NAT/iptables rule at container start. EXPOSE/-P is documentation, not a live mechanism. There’s no API for a running container to add another published port — the container has to be recreated with a new -p list.
• Kubernetes: a Service’s spec.ports is a fixed list, and a container’s own ports: field in the Pod spec is documentation only (Kubernetes doesn’t firewall ports you didn’t declare, and it doesn’t publish ones you didn’t declare either). Nothing watches a container’s live sockets and reconciles a Service to match them.

Two patterns actually work, and wayhouse already has one foot in each:

1. Host networking (docker run --network host / hostNetwork: true on the Pod) — the container shares the host’s network namespace directly, so every bind() wayhouse does is immediately live with zero orchestration involvement. This is the closest match to wayhouse’s own reload-driven listener reconciliation: add a bind: entry, send SIGHUP (or let file-watch pick it up), and the new listener exists — in a container exactly like bare metal. Cost: no network-namespace isolation for that pod/container — an acceptable, often expected, trade-off for the thing that sits at the actual network edge.
2. A pre-reserved port range — publish/expose a large contiguous range up front (Docker supports a range: -p 20000-40000:20000-40000/udp; Kubernetes needs either hostNetwork scoped to a node-level reserved range, or enumerating hostPort per container — there’s no native range syntax in a Pod spec) and only actually bind() the subset needed at any given time via wayhouse’s own bind: "host:lo-hi" range listener. This is the same shape Agones (the standard Kubernetes game-server operator) uses for its GameServer pods — hostNetwork: true plus one port pulled from a node-level reserved pool — because Kubernetes Services can’t do dynamic port publishing either; docs/08’s roadmap entry for the range-bind feature draws this comparison explicitly.

Host networking is the simpler default when the proxy doesn’t need network-namespace isolation from other workloads on the node (usually true for an edge process). Reach for the reserved-range pattern when isolation is a hard requirement, or when running many wayhouse instances per node and the orchestrator should own port allocation/collision-avoidance across them.

--tunnel-* (phase 14): CAP_NET_ADMIN and /dev/net/tun

wayhouse --tunnel-* and wayhouse-agent create a real WireGuard interface (docs/11) — kernel-netlink by default, falling back to boringtun userspace with --tunnel-userspace/--userspace. Both backends need CAP_NET_ADMIN, and the userspace fallback additionally needs /dev/net/tun present in the container. Grant them explicitly:

docker run --user 0 --cap-add=NET_ADMIN --device=/dev/net/tun ...

--user 0 matters for the nonroot images in deploy/: Docker and Kubernetes put an added capability only in a non-root process’s bounding set, so uid 65532 would still get EPERM creating the interface.

securityContext:
  runAsUser: 0
  runAsNonRoot: false
  capabilities:
    add: ["NET_ADMIN"]
volumes:
  - name: tun
    hostPath: { path: /dev/net/tun }

This is a real, already-verified requirement, not a guess: docs/08 phase 14 slice 6’s end-to-end verification ran in exactly this container shape (crates/wayhouse-agent + wayhouse --tunnel-* in Docker containers with NET_ADMIN + /dev/net/tun) because the project’s own dev sandbox lacked CAP_NET_ADMIN even in the bounding set. A container without both of these fails loud at interface bring-up, before any listener binds or backend source is built — it does not silently run without tunnel support.

The CI-runnable make tunnel-e2e (rootless network namespaces, docs/superpowers/specs/2026-10-01-tunnel-e2e-design.md) verifies the tunnel logic with the same binaries and flags, but it does not exercise container capabilities — NET_ADMIN and /dev/net/tun as documented above stay the deployment requirement.

Authentication at startup

Every fleet API is open when no token is configured. Startup now enforces:

• Non-loopback bind needs auth. wayhouse-controller, wayhouse-aggregator and wayhouse-ui (--listen) and wayhouse’s admin API (settings.admin.listen) refuse to start on a non-loopback address without a token (--auth-token, --ui-password / --users-file, settings.admin.auth_token). If the network boundary really is your only control, pass --insecure-no-auth; it logs a warning instead.
• HA needs --ha-token. wayhouse-controller --ha-peers / --ha-join refuse to start without it, with no opt-out: /raft/* and /admin/ha/members would otherwise accept anyone. HA peers should use https:// URLs (see below), since the token travels in each request.
• Minimum secret length: 16 bytes for --auth-token, --ha-token, settings.admin.auth_token and settings.gossip.psk. Generate one with openssl rand -hex 32. --ui-password and the outbound tokens (--controller-token etc., which must match the remote side) are not checked.

TLS for the fleet services

Without the settings below, wayhouse-controller, wayhouse-aggregator, wayhouse-ui and wayhouse’s admin API serve plain HTTP. Run as-is across a network, that exposes:

• the --auth-token bearer token on every request, and the /admin/adopt calls;
• the full config text, on GET /config and the SSE GET /config/subscribe;
• every origin’s registration (WireGuard public key, public endpoint, fronted backend addresses) on the /peers* and /proxy-peers* routes.

Two ways to encrypt it: native TLS (below), or a reverse proxy you run that terminates TLS, with the controller listening only on loopback or a private network (--listen 127.0.0.1:9901, or a private bridge/pod network). The aggregator, the UI and wayhouse’s admin API serve native TLS the same way.

Native TLS

wayhouse-controller --listen 0.0.0.0:8443 \
  --tls-cert /etc/wayhouse/tls/fullchain.pem --tls-key /etc/wayhouse/tls/privkey.pem

• --tls-cert is a PEM chain, leaf first (a Let’s Encrypt fullchain.pem works); --tls-key a PEM private key (PKCS#8, SEC1 or PKCS#1). Both or neither. With them, --listen speaks HTTPS only — every route, including the SSE streams.
• Renewal without a restart: both files are checked every 30 s and a changed pair is swapped in for new connections. A broken or half-written pair (no certificate, no key, key not matching the certificate) is logged and the current certificate stays in service until a good pair appears. Bad files at startup stop the controller with an error naming the file.
• Clients use https:// URLs; with a private CA they add --ca-file. HA replicas can each serve native TLS and name their peers --ha-peers 1=https://ctl-1:8443,… (plus --ca-file), with no terminator at all.
• TLS handshakes run in their own tasks, so a client that connects and stalls cannot hold up others. A client must send its ClientHello within 3 s and finish the handshake within 10 s. At most 16 handshakes may be pending per source (an IPv4 address or an IPv6 /64); more are closed at once. At 512 pending handshakes in total a new connection is still accepted and the oldest pending handshake is dropped, so a flood of idle connects cannot lock real clients out. A source may also open at most 20 new connections a second on average (bursts of 64), so it cannot cycle connects under its pending cap. A limit being hit logs a warning (at most once a minute) and counts into wayhouse_tls_handshakes_refused_total / wayhouse_tls_handshakes_evicted_total on every binary’s /metrics (docs/06). The limits are flags: --tls-max-pending (512), --tls-max-pending-per-source (16), --tls-new-per-source-per-sec (20; 0 turns the rate limit off) and --tls-new-per-source-burst (64); raise the per-source ones for a fleet that reaches the service from behind one NAT. No client certificates (mTLS).
• wayhouse-aggregator takes the same two flags with the same behaviour. Instances then push to --aggregator https://… (plus --ca-file for a private CA), and the same goes for a child tier’s --parent-url and wayhouse-ui --aggregator-url.
• wayhouse-ui takes the same two flags. Serving HTTPS itself, it marks the session cookie Secure (without the flags it doesn’t — a browser drops a Secure cookie on http:// and login would loop). The live view’s WebSocket works over HTTP/2 too: a browser on h2 opens it as an extended CONNECT, which the UI accepts (and the session cookie counts in whichever cookie header h2 splits it into). Plain HTTP is not redirected; serve only HTTPS on the port browsers use.
• wayhouse’s admin API is configured in the YAML, next to listen: settings.admin.tls: { cert: <chain.pem>, key: <key.pem> } (both required, plus the optional handshake limits max_pending, max_pending_per_source, new_per_source_per_sec, new_per_source_burst, same meaning as the flags above; startup-only like listen; the files renew like the flags above; wayhouse --check loads them). The admin URL wayhouse reports to the aggregator then becomes https://<settings.admin.listen>, so the certificate needs that address as a SAN (an IP SAN for an IP listen), and the aggregator takes --ca-file for a private CA. As before, the reported URL is the literal listen address — a wildcard bind (0.0.0.0) is not reachable as an admin URL, TLS or not. With --controller one YAML reaches every instance, so put each host’s own certificate at the same path. Health probes against the admin port must then use HTTPS (Kubernetes: httpGet.scheme: HTTPS).

Verified by wayhouse-fleet-tests: controller_native_tls.rs (wayhouse --check fails on UnknownIssuer without --ca-file and passes with it; /config/subscribe streams over TLS; --tls-cert alone is refused), aggregator_native_tls.rs (a wayhouse pushes to an https:// aggregator, read back over HTTPS), ui_native_tls.rs (login over HTTPS sets a Secure cookie that unlocks /ui/session), admin_native_tls.rs (an aggregator fans an intent verb out to an https:// admin API; wayhouse --check names a bad settings.admin.tls.cert) and ha_tls.rs three_replicas_replicate_over_native_tls; certificate loading, renewal and the listener are unit-tested in crates/wayhouse-http/tests/tls_{certs,server}.rs.

Proxy configuration

Four routes are long-lived server-sent-event streams — /config/subscribe, /peers/subscribe, /proxy-peers/subscribe and /intent/subscribe. The controller sends keep-alives, but a proxy that buffers responses or enforces a short read timeout will stall or drop them, so those must be off/long.

Caddy (certificates from Let’s Encrypt automatically):

controller.example.com {
    reverse_proxy 127.0.0.1:9901 {
        flush_interval -1        # stream SSE immediately
    }
}

nginx:

server {
    listen 443 ssl;
    http2 on;
    server_name controller.example.com;
    ssl_certificate     /etc/letsencrypt/live/controller.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/controller.example.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:9901;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_buffering off;      # SSE
        proxy_read_timeout 1h;    # idle SSE streams
    }
}

These two snippets are not tested in this repository (CI has neither proxy, and an end-to-end https test would need a certificate the clients trust — see the limits below). Check them against your proxy’s version.

In Kubernetes the equivalent is an Ingress (or Gateway) with TLS in front of the wayhouse-controller Service; make sure its response buffering and timeouts allow SSE.

Pointing the clients at it

Every HTTP client in the fleet uses reqwest with rustls and builds its requests from the base URL as given — no code forces a scheme — so these take an https:// base URL: wayhouse --controller, wayhouse --tunnel-controller-url, wayhouse-agent --controller-url, wayhouse-ui --controller-url / --aggregator-url, wayhouse-controller --parent-url, and the parent_url in a POST /admin/adopt body. To check it works:

curl -fsS https://controller.example.com/healthz
wayhouse --check --controller https://controller.example.com --controller-token "$TOKEN"

Private or self-signed CA. Every binary (wayhouse, wayhouse-agent, wayhouse-controller, wayhouse-aggregator, wayhouse-ui) takes --ca-file <PATH>: a PEM file with one or more CA certificates to trust for all of that process’s outbound HTTPS, in addition to the built-in Mozilla roots (it never replaces them, so public endpoints keep working). Non-certificate PEM sections (a private key pasted into the same file) are ignored. A missing or unreadable file, one with no certificates, or a malformed certificate stops the process at startup with an error naming the file. The file is read once, at startup: SIGHUP / a config reload does not re-read it, so restart the process after changing it (e.g. when rotating the CA).

wayhouse --check --controller https://controller.internal:8443 --ca-file /etc/wayhouse/ca.pem

This exact shape — wayhouse --check against a real wayhouse-controller behind a TLS terminator whose certificate a private CA signed, failing without --ca-file and passing with it — is the wayhouse-fleet-tests test ca_file.rs, run by make check. The two proxy snippets above are still untested.

A TLS or connection failure is logged with its cause (e.g. invalid peer certificate: UnknownIssuer), not just “error sending request” — that is usually the first sign a --ca-file is missing or wrong.

HA replicas over TLS

Put a TLS terminator in front of each replica’s listener and give --ha-peers base URLs instead of host:port (same list on every replica), plus --ca-file if the terminators’ certificates come from a private CA:

wayhouse-controller --listen 127.0.0.1:9901 --ha-node-id 1 --ca-file /etc/wayhouse/ca.pem \
  --ha-peers 1=https://ctl-1.internal:8443,2=https://ctl-2.internal:8443,3=https://ctl-3.internal:8443

Raft RPCs (/raft/append, /raft/vote, /raft/snapshot) and writes a follower forwards to the leader then go over HTTPS. id=host:port still means plain HTTP, and the two forms can be mixed. An entry that is not http(s)://host[:port] (another scheme, a path, a query, user info) stops the replica at startup naming the entry. Verified by wayhouse-fleet-tests ha_tls.rs: three replicas that reach each other only through private-CA terminators elect a leader, forward writes and all serve the last one with --ca-file, and never elect a leader without it.

Only at bootstrap. --ha-peers is read when the cluster is first initialised; after that each member’s address lives in the replicated Raft membership. Changing --ha-peers on an existing cluster does not change the addresses replicas use. To move a running cluster to https:// peers, change each member’s address with a PUT (see “HA membership” below).

HA membership

Members are changed on a running cluster with the admin routes, gated by --auth-token (pass it as a bearer token) and answered by any node, which forwards a change to the leader. X-Actor is logged.

# A new or replacement node always starts empty with --ha-join (never --ha-peers: a
# node with --ha-peers calls initialize and could split the cluster).
wayhouse-controller --listen 0.0.0.0:7070 --ha-node-id 4 --ha-token "$HA_TOKEN" \
  --tunnel-network 10.60.0.0/24 --ha-join

curl -H "Authorization: Bearer $TOKEN" http://ctl-1:7070/admin/ha/members            # voters, learners, leader
curl -H "Authorization: Bearer $TOKEN" -d '{"id":4,"addr":"ctl-4:7070"}' \
  http://ctl-1:7070/admin/ha/members                                                # add (waits for catch-up)
curl -H "Authorization: Bearer $TOKEN" -X DELETE http://ctl-1:7070/admin/ha/members/2  # remove
curl -H "Authorization: Bearer $TOKEN" -X PUT -d '{"addr":"https://ctl-3.internal:8443"}' \
  http://ctl-1:7070/admin/ha/members/3                                              # change an address

An add answers once the new node has caught up, by log or, after the log was purged, by snapshot; a large catch-up can take minutes (a forwarded add waits up to five). Before an add or a PUT the leader asks the node at the given address who it is (/raft/whoami, behind --ha-token) and refuses (422) unless it answers with that node id, and an add also unless the node’s log is empty (or it is already a learner of this cluster): this keeps a node id from being pointed at another node’s address, which openraft warns can produce two leaders. Other answers: 409 the node is already a voter, 404 unknown id, 422 removing the last voter, 503 the node is unreachable or there is no leader. Removing the current leader is allowed. A joined node restarts with --ha-join again. --ha-join and --ha-peers together, and --tunnel-readdress with either, are refused at startup. Verified by wayhouse-fleet-tests ha_tunnel_addresses.rs (join, remove the leader, move a node to a new port) and ha_snapshots.rs (catch-up by snapshot after a purge).

Upgrading a single controller to HA

A controller that already holds registrations (origins, proxies, addresses) from before HA can become the first node of an HA cluster. Restart it with --ha-node-id and the --ha-peers of the new cluster next to the other, empty nodes. Every node sets aside pre-HA data it finds as peers.pre-ha/, proxy-peers.pre-ha/ and tunnel-addresses.pre-ha/ under --data-dir (it never deletes them), and the cluster’s leader imports exactly one node’s copy, whichever node wins the first election.

• The leader asks every voter what it holds (/raft/whoami) before initializing, so registry writes answer 503 until every voter has answered; a voter that is down delays the upgrade. --ha-import-source <node-id> (the same value on every node) waits only for that node; none imports nothing.
• When more than one node has pre-HA data (for example a pin-only --ha-peers deployment where each node kept its own registries) nothing is initialized: the leader logs an ERROR naming the nodes and their counts, and registry writes stay 503 until you restart the nodes with --ha-import-source <node-id>. The other nodes’ set-aside data stays untouched (one WARN each).
• Imported registrations are re-written as new revisions that continue after the source’s last revision number, so every edge’s since cursor from the single-node days stays valid. Tombstones the old node wrote are carried over too (the entry lists the names whose newest log entry is a removal), so a subscribed edge that had not received one still drops the peer, and a registry with no live registration keeps its log head. ImportContent is a Raft log entry: a node on an older build applies an Import without those tombstones, so upgrade every node of the cluster before the first initialization.
• Clients’ pinned addresses that used to collide across nodes now get 409: uniqueness is cluster-wide, where it was per node before.
• Snapshots persisted by a build older than this feature (before 2026-10-03) are not supported: a follower refuses to install one (ERROR naming the snapshot format) rather than read it as empty registries and an empty book. Start such a node from empty storage, or let the cluster re-initialize as above.
• The registries compact their logs: each keeps one entry per name (its registration, or the tombstone of a removed one), so a registry log and the Raft snapshot stay bounded by the number of distinct names ever registered, not by how often they re-register. Revision numbers have gaps afterwards; a subscriber’s since cursor is unaffected.

Limits (read these before relying on it)

• No system certificate store. The clients trust the Mozilla root bundle compiled into the binary (reqwest’s rustls-tls / webpki-roots) plus --ca-file, not the OS store or SSL_CERT_FILE. For a private or internal CA — or a self-signed certificate — pass it with --ca-file. No client certificates (mTLS).
• HA over TLS (native TLS on each replica, or a terminator per replica; see “HA replicas over TLS”): bootstrap with https:// peers, or move a running cluster with one PUT /admin/ha/members/{id} per member (see “HA membership”). With host:port peers, replica traffic is plain HTTP; then keep the replicas on a private network — --ha-token is a shared secret, not encryption. The controller logs a warning for each non-loopback http:// peer.
• Upgrading a gossip mesh (the datagram format gained a MAC’d sender timestamp, security review O4): upgrade all instances in a failure domain together. While old and new instances are mixed they cannot exchange membership or health, so the domain quorum derived from gossip is unreliable until the rollout finishes (local health checks keep working). Old-to-new datagrams are counted in wayhouse_gossip_stale_rejected_total. Instance clocks must also agree within 30 s.
• The UI behind a proxy sees plain HTTP, so its session cookie is HttpOnly; SameSite=Lax but not Secure: add it, redirect HTTP to HTTPS (and consider HSTS) at the proxy — or use the UI’s native TLS, which sets Secure itself.
• A TLS proxy is a trust boundary. It sees every token and registration in the clear. Run it on a host you control — or use native TLS where it exists.

Certificates from the controller

With --acme the controller can issue the web front door’s certificates; a gateway host fetches them with wayhouse-certsync, which ships in the wayhouse image. See acme.md for the gateway token, the files it writes and how to run it.

Tunnel addressing

Tunnel-internal addressing (the WireGuard-side space between proxies and origins) is not an open question any more: wayhouse-controller --tunnel-network allocates the addresses, so nothing here hand-picks one. See docs/11 “Address authority”.

Choosing the network. Use an IPv6 unique local address (ULA) network by default: generate a random /48 once and take a /64 from it, so it cannot clash with any origin’s own LAN or another deployment’s tunnel:

printf 'fd%02x:%02x%02x:%02x%02x::/48\n' $(od -An -N5 -tu1 /dev/urandom)
# e.g. fd49:89c1:4b5e::/48 -> --tunnel-network fd49:89c1:4b5e:60::/64

Choose an IPv4 network (/16 to /30, e.g. 10.60.0.0/16) instead when a game server binds 0.0.0.0 only: it cannot be reached on an IPv6 tunnel address. The family is fixed per controller; changing it later needs --tunnel-readdress (docs/11).

Containers. The tunnel interface needs IPv6 enabled in its network namespace. Docker starts containers on a network without IPv6 with net.ipv6.conf.all.disable_ipv6=1; for a container with its own network namespace set sysctls: { net.ipv6.conf.all.disable_ipv6: "0" } (Compose) or the equivalent securityContext.sysctls entry (Kubernetes). With host networking, as in deploy/compose/compose.tunnel.yml and the DaemonSet’s hostNetwork, the host’s own setting applies and such a sysctl is refused, so the host must not have IPv6 disabled.

This document otherwise covers only the container/orchestrator-facing side of the proxy’s public listeners.

Upgrading a fleet

How to move a running deployment to a newer version one component at a time, without a flag day. The design and its reasoning are in the component versioning spec; this page is the operator’s view of what is built. Releases and tags are in RELEASING.md.

Compatibility rules

• Window: N and N-1. Two adjacent minor versions of the product (0.N and 0.(N-1)) are meant to run side by side during an upgrade. While the product is 0.x, “minor” is the second number.
• Protocol major must match. A node that receives another major (or an unreadable X-Wayhouse-Protocol header) answers 426 and counts it in wayhouse_protocol_mismatch_total. A breaking wire change bumps the major and, to keep the window honest, must ship with a gate that also admits the previous major for one product minor and routes those requests to a compatibility shim; every other major still gets 426. Today the gate admits only the current major and no shim exists, because no second major exists yet.
• Protocol minors are additive. A sender only uses a newer optional field or route with a peer that has said it understands it: every request carries the caller’s version and every response the server’s, and a peer whose version is not known yet gets the baseline. Receivers ignore unknown JSON fields.
• Config is the exception to “controllers first”: the config schema_version is the minimum schema a document needs, and the controller refuses (422) a document newer than the lowest max_config_schema any live proxy registration reported. Upgrade every proxy before you use a config field the new version introduced.
• Store: a controller refuses to open a store written by a newer build and migrates an older one once, on open.

What is and is not checked: the header check enforces protocol majors only. The N / N-1 product window itself is a rule kept by review (see the AGENTS.md row for wire changes) and exercised by tests that inject an older protocol minor into a real fleet (crates/wayhouse-fleet-tests/tests/mixed_versions.rs). Once two releases exist, the non-required compat workflow also runs the previous release’s images against the current ones.

Version table

One row per release, newest first. A CI check (.github/scripts/check_upgrading_doc.py) fails when the first row disagrees with the constants in the source, so a protocol, schema, store or ABI change cannot land without its row. Use unreleased for the product column until the release is tagged.

Upgrading from 0.1.0 to 0.2.0

0.2.0 is the first release with a real upgrade step. Things to know before you start:

• Gossip does not cross the version. 0.2.0 gossip frames carry a magic byte and GOSSIP_VERSION; a 0.1.0 node and a 0.2.0 node drop each other’s datagrams. While a fleet runs both, health votes and domain quorum are not shared between old and new nodes, and each node falls back to its own checks. Existing player traffic is not affected, but nodes can briefly disagree on backend health, so new sessions may be placed differently on old and new nodes. This is the one exception to the N and N-1 window (see docs/10).
• Registry names. A name must be 1 to 64 characters of A-Za-z0-9._- and not subscribe; see “Dedupe first” below for the check to run before the first replica is upgraded.
• Bundled sniffers moved. The official WASM sniffers now live in wayhouse-proxy/sniffers and are no longer built from this repository. Get them from that repository’s releases, install them from the UI’s registry page, or copy them into the directory named by settings.sniffers.dir (see sniffers.md). sniffers.lock in this repository only pins the releases the CI loader tests run against; it has no effect on a deployment.
• Config schema 2. listeners.plugin_routes is a schema 2 field. A 0.1.0 proxy cannot parse it, so use it only after every proxy runs 0.2.0. The controller enforces this on submit (422 naming the lowest live proxy maximum). A proxy that does not announce a schema (0.1.0) is sent only revisions that need schema 1: one that first starts after the first schema 2 revision exists gets the newest schema 1 revision, not the newer ones, and keeps it until it is upgraded.
• Plugin ABI. The version table has no plugin column: the plugin ABI is 0.1 and the host requires an exact minor match while the major is 0.

Order of operations

1. Aggregator and UI (the read side).
2. Controller tiers, leaf slave tiers first and the root tier last. A slave tier is a child controller under a parent. Inside the window either order works because of the gating rule above; going leaves-first keeps the root, which owns writes, on the version your operators have run longest.
3. Raft HA inside one tier, see below. A tier made of several controllers is a different mechanism from the parent and child tiers.
4. Proxies and agents, a few at a time: drain, upgrade, rejoin.
5. Config fields introduced by the new version, only after every proxy runs it.

Check the fleet view in the UI between steps: an instance more than one minor behind the newest, on another protocol major, or that refused a request for an incompatible protocol in the last five minutes is red; one minor behind is yellow.

The fleet view lists only the instances that push to the aggregator, which are the proxies. Controllers, the aggregator, the UI and agents do not appear in it. For those, read wayhouse_build_info on each component’s /metrics between steps (its version and protocol labels).

Controller tiers

Upgrade a tier completely (all its replicas, see below) before moving to its parent. Parent-to-child and child-to-parent calls both carry protocol versions, so a child one minor older than its parent is sent only baseline fields. A slave tier relays writes upward through its leader; a tier mid-upgrade keeps relaying as long as the protocol major matches.

The baseline-fields rule for a child one minor older is not covered by an automated test for the tier path (the fleet tests exercise a proxy against a controller, not a slave tier against its parent), and neither is the relay cursor across versions (a child resuming the parent’s revision stream after one side upgraded). Treat the first tier upgrade in your environment as a rehearsal, and read the “N and N-1” window above as tested for proxy against controller only.

Raft HA inside a tier

Replicas of one tier replicate through Raft; /raft/* carries the same protocol header check, and changing a raft RPC payload counts as a breaking wire change.

1. Upgrade the followers one at a time. Wait for the upgraded node to catch up (GET /admin/ha/members shows it as a voter again) before touching the next.
2. There is no leader-transfer command. To move leadership, stop the leader process; the followers elect a new leader within the election timeout and writes resume through it.
3. Upgrade the former leader last.

Dedupe first. Since 0.2.0 a tier’s replicated registries refuse a registration whose WireGuard public key another name already holds (409). That check runs when a log entry is applied, so a 0.2.0 node can reject an entry that a 0.1.0 node accepted, and a fresh node replaying an old log does the same; the replicas would then hold different registry contents. Before the first replica is upgraded, list GET /peers and GET /proxy-peers on the tier and DELETE every extra name that shares a public key with another, keeping the one that is in use. Also rename every name outside [A-Za-z0-9._-], longer than 64 bytes, or equal to subscribe: since 0.2.0 a new registration with such a name is refused at submit time (422). Entries already in the log are not revalidated when applied, but a node whose registration refresh is refused under 0.2.0 drops out of the registry.

This sequence is not covered by an automated test (the fleet tests do not run a rolling Raft upgrade); treat the first use in your environment as a rehearsal. A write that a follower forwards to the leader is sent with the follower’s protocol version, not the original caller’s; that is harmless inside the window because receivers ignore fields they do not know.

Proxies

A proxy upgrade relies on draining:

1. POST /admin/drain: the instance reports not ready, so the load balancer or anycast routes new clients elsewhere. The data path keeps running.
2. Wait for sessions to end (active_conns in GET /config, or let shutdown_grace_sec expire).
3. Stop the old process, start the new one, check GET /readyz, and let the load balancer bring it back.

A config reload that replaces a UDP listener no longer drops new flows while the old group drains (#186), but it ends that listener’s live sessions: the old workers close their sockets at once, and each client opens a fresh session in the replacement group on its next datagram (possibly from a new upstream source port). Plan a reload that changes a UDP listener like a short blip for its players. A listener that is removed, or whose bind address changed, still drains as described above. See A config reload that replaces a UDP listener.

Agents

Agents register with the controller like proxies do. Upgrade them after the controller tier they register with; an agent one minor older than its controller works against baseline fields.

Kubernetes

The same order applies, with a rolling update (maxUnavailable: 1) and the readiness probe gating each step. Pending #88: the packaged manifests for this do not exist yet, so this section describes the intent, not a tested procedure.

Rollback

Roll back in the reverse order. A downgrade is safe inside the window for the protocol. A controller store that a newer build has migrated cannot be opened by an older build (FormatTooNew): restore the store from a backup taken before the upgrade, or keep the upgraded controller. Config documents that use a field of the newer schema are rejected by older nodes, so revert the controller to the previous config revision first.

Troubleshooting

Plugins

A plugin is a WASM module that reacts to a trigger and acts through capabilities the operator approved (design: plugin system and automation hooks). Plugins are not sniffers: sniffers run on the proxy data path.

Status. The host runtime is built (crates/wayhouse-plugin-host) and a standalone controller can store plugin installs behind an admin API and ticks the enabled ones (below); the web UI has a Plugins page on top of the install API (see “The Plugins page”). An HA tier replicates installs and state through Raft, serves the same API on every replica and ticks on the leader only, and the module bytes follow the installs to every replica (see “High availability”). Secrets, http, routes, webhooks and events are built; backends is not.

Module contract (ABI 0.1)

Link wayhouse-plugin-abi (it stamps the wayhouse.plugin-abi section and exports alloc). The host accepts a module only when the section equals its own version (exact minor match while the major is 0), and reads the section before compiling anything.

A second custom section, wayhouse.plugin-caps, holds the capabilities as JSON. It is read without running the module and approvals are recorded against the module’s sha256:

{
    "triggers": { "on_timer": true },
    "tick_interval_secs": 30,
    "log": true,
    "state": { "max_bytes": 65536 },
    "http": {
        "hosts": [
            { "host": "panel.example", "port": 8443 },
            { "host": "10.0.0.5", "allow_private": true }
        ]
    },
    "secrets": [{ "name": "PANEL_TOKEN", "hosts": ["panel.example"] }]
}

Unknown fields are an error. tick_interval_secs is required with on_timer and at least 10 and at most 604800 (one week).

Exports: memory, alloc(i32) -> i32, init(config_ptr, config_len), and on_timer() when declared. The only imports allowed are these, in the wayhouse namespace:

| routes_set(ptr, len) -> i32 | routes | declares the plugin’s whole route set as a JSON array of {"host", "backend"}; 0, or -1 when refused (the reason is a warn line in the plugin’s log) | | acme_dns_result(ok, ptr, len) -> i32 | on_acme_dns | answers an on_acme_dns call: ok 1 or 0 and a message of at most 512 bytes (UTF-8); 0, or -1 when ok is neither or the message is too long; the last call wins | | http_request(ptr, len) -> i32 | http | sends the JSON request at ptr; returns the length of the response document, or -1 when the request is over the size limit | | http_read(out_ptr, out_cap) -> i32 | http | copies the pending response document out; -1 (and keeps it) when out_cap is too small |

A call that uses a capability it did not declare traps that call (CallError::CapabilityDenied) and leaves the host running.

Secrets and the http capability

secrets declares slots by name (an upper-case letter then A-Z, digits, _; up to 64 characters), each bound to hosts that the module also lists under http.hosts; the operator approves the set with the module’s sha256 like any other capability. http.hosts entries are exact lower-case host names or IP literals, with an optional port (default 443); allow_private must be approved per host to reach private, loopback or link-local addresses.

A slot’s hosts entries are host or host:port ([v6]:port for an IPv6 literal). A host:port entry binds the slot to that approved port only. A bare host binds it to the host’s one approved port, and a module that lists the host on several ports must name one (panel.example:8443), so a secret never reaches a second port of the same host by accident.

The request. http_request takes UTF-8 JSON, {"method", "url", "headers", "body"} (body base64, optional), and http_read returns {"status", "headers", "body"} (body base64), or {"error": "..."} for a refused or failed call. The host enforces: https only; a host and port the module declared; no proxy; no redirect to anywhere but another approved host (up to 3 hops, each re-checked); at most 8 calls, 256 KiB of request and 1 MiB of response body per call, 10 s per request and 20 s per guest call; a fixed header denylist (Host, Content-Length, …). Connecting resolves the name once and checks every resolved address, so a DNS answer that changes cannot reach a private address that was not approved; an IP literal is checked the same way.

Secrets never reach the guest. A header value may contain ${secret:NAME}. The host expands it, only in header values (not in the URL or the body) and only when the request goes to a host the slot is bound to, and re-checks that for each redirect hop. The response (headers, body, error text) and every log line are scrubbed of the secret’s raw, base64 and percent-encoded forms before the guest or the status sees them.

Storage. Operators set a slot with PUT /plugins/{id}/secrets/{slot} and {"value": "..."} (16 to 4096 bytes), clear it with DELETE and list the slots with GET /plugins/{id}/secrets (set or not, the key id, ok or held: ...; never a value). Only approved slots are accepted. A value is encrypted on the controller that received it with XChaCha20-Poly1305 under a cluster keyring, bound to install, slot and key id, and only the ciphertext goes into the sled database, the Raft log and snapshots. A follower seals the value itself and sends the ciphertext entry to the leader (POST /raft/plugin-secret); it never forwards the plaintext request. Deleting an install deletes its secrets. Changes are audit-logged (install, slot, key id, actor; never the value) at target wayhouse_controller::plugins::audit, and a delete or rewrap asks every node to purge the log, so old ciphertext does not outlive it for long.

Keys. --plugin-secret-key-file (or WAYHOUSE_PLUGIN_SECRET_KEY) names a keyring: one base64 key of 32 random bytes per line (openssl rand -base64 32), the last line active, the file readable by its owner only. Every controller of a tier needs the same keyring, and the file is re-read when it changes. Without a key the secrets API answers 409 and a plugin that uses secret slots is held: it is not run, GET /plugins/{id}/status shows held: ... (no key, secret unset, key missing, or not authenticating), and other plugins keep running. Plaintext is kept out of the store and the log, and the controller wipes the buffers it owns, but this is best effort: the HTTP stack copies a request body and an expanded header value into memory this project does not control (HeaderValue, the PUT body Bytes), and those copies are not zeroed. Anyone who can read the controller’s or the proxy’s memory can read live secrets. Plaintext HTTP on a non-loopback --listen refuses to start with --plugins and a key unless --allow-insecure-secrets is given.

Rotating a key. (1) Append the new key to the keyring file on every controller. (2) POST /admin/plugins/secrets/rewrap on any controller: it refuses until every member reports the active key (GET /admin/plugins/secrets/keys lists the key ids each holds), then re-encrypts every secret under it; a slot an operator rewrote meanwhile is skipped and reported. A member that does not answer is skipped, so a rewrap also works while a replica is down; that replica catches up on its own once it is back, and until it has the new key in its keyring the plugins that use it are held. (3) Remove the old key from the file. Removing it before step 2 holds the plugins that still use it (held: key <id> missing) until it is put back.

Routes

The routes capability lets a plugin publish hostname routes:

"routes": { "hosts": ["*.mc.example.com", "play.example.com"], "backends": ["10.0.0.0/16"], "max_entries": 64 }

hosts are the names (exact or *.suffix) the plugin may claim, backends the networks its backend addresses must be in, max_entries (1 to 256) the size of one set. A module cannot ask for more of any of these than was approved. routes_set replaces the plugin’s whole set; the host refuses a set with an entry outside the approval, a duplicate hostname or too many entries, and a tick that does not call it leaves the set as it was. The set is committed in the same entry as the tick’s state writes (term and revision checked, validated again on every replica), and only when it changed.

Materialisation (#236). The controller does not rewrite the operator’s config. It publishes the routes of every enabled install as an overlay (GET /plugin-routes, and a stream at GET /plugin-routes/subscribe that proxies follow), and a proxy started with --controller appends them to each TCP listener that opted in with plugin_routes: (config schema 2, see docs/05-configuration.md). The listener says how the hostname is read (sni, or a sniffer). Rules:

• Plugin routes go after the listener’s own routes, so operator routes always win, including an always fallback.
• Each distinct backend address becomes a one-target pool named plugin-<hash>.
• Two installs claiming the same hostname: the earlier install wins; the later claim is not applied and is listed under conflicts in GET /plugin-routes, with owner (plugin:<install id>) and plugin name on every route.
• Exact names match before *.suffix patterns, longer suffixes first.
• Disabling or deleting an install withdraws its routes at once. A plugin whose ticks fail keeps its last committed set, and so does one that stops calling routes_set: the set stays until the next successful tick replaces it or the install is deleted or disabled. (The spec’s grace period with an alert is not built; see the follow-up issue.)
• A controller started without --plugins answers 501 (404 on an older build) and proxies drop the plugin routes at once, keeping their own config; a controller that is only unreachable leaves the last overlay in place. A proxy without --controller ignores the overlay.

Webhooks

A plugin that declares "triggers": {"on_webhook": true} can be called by an outside system. Webhooks are off until the operator starts the controller with a separate listener, never on the admin port:

--plugin-webhook-listen 0.0.0.0:8444 --plugin-webhook-tls-cert hooks.pem --plugin-webhook-tls-key hooks.key

It needs --plugins. Plain HTTP on a non-loopback address is refused at start unless --allow-insecure-secrets is given (the token and body would cross the network in clear text).

Enabling. POST /plugins/{id}/webhook (admin) creates a token for an install that was approved for on_webhook and answers {"token": "whk_...", "path": "/plugins/{id}/hook"}. The token is shown once; only its SHA-256 is stored and replicated. Calling it again rotates the token (the old one stops working at once); DELETE /plugins/{id}/webhook turns the webhook off. Under HA the call needs every member to run a build that supports it.

Calling. POST <listener>/plugins/{id}/hook[/suffix][?query] with Authorization: Bearer <token>. In order:

1. A per-source rate limit (IPv6 by /64) runs first, valid token or not: 429. The source is the TCP peer address. Callers behind one load balancer or NAT therefore share a single bucket, and the limit applies to all of them together; there is no trusted-proxy header. Put the webhook listener where callers reach it directly; there is no setting to relax the source limit.
2. The token is checked in constant time, against a dummy hash when the install is unknown, disabled or has no webhook. Every failure is the same 401.
3. A per-install rate limit: 429.
4. Only then the body is read, capped at 256 KiB: 413.
5. The call runs as one of the install’s serialized calls (at most 4 hook calls wait per install; the timer is never rejected). A full queue is 429 and is counted in the status (hooks_dropped); a plugin that fails is 502.

The guest receives the method, suffix, query, headers (never Authorization or Cookie), the body (base64) and an idempotency key (the Idempotency-Key header, else a hash of the install and body). It answers with webhook_respond (status, headers, base64 body); only plain headers are passed on, and the response is never cached. The call’s state writes are committed before the answer goes out, so a 2xx means the state is durable; a commit that fails is 503 and the caller should retry (use the idempotency key to make that safe).

HA. Any node may receive the request. A follower authenticates it itself and forwards it to the Raft leader over the peer channel (/raft/plugin-hook/{id}, peer token, protocol gate), but only over https peers: with plain-HTTP peers it answers 503 rather than send the body in clear text. Run the webhook listener on every node you want to take requests, or point the sender at one.

Events

A plugin that declares "triggers": {"on_event": ["config_revision", "plugin_changed"]} (and on_timer, which stays required) gets on_event(kind, payload) calls:

Events are hints, not a log. They use the same bounded hook queue as webhooks (at most 4 waiting per install), so a slow plugin loses events instead of building a backlog; the loss is counted as hooks_dropped in the status. A restart, a failover (only the Raft leader delivers) or a lagging feed also loses some. The list of kinds only grows. A plugin that must not miss a change reconciles from the controller’s state on its timer and treats an event as a reason to do it sooner. A backend-health event is not built yet.

ACME DNS-01 (on_acme_dns)

A DNS-provider plugin declares "triggers": {"on_timer": true, "on_acme_dns": true} (the timer stays required, like for events), exports on_acme_dns(ptr, len) and answers with the acme_dns_result import. The controller’s ACME runner calls it on the leader to set and remove the TXT record a CA validates (acme.md). The request document is:

{
    "op": "present",
    "fqdn": "_acme-challenge.panel.example.com",
    "value": "<txt value>",
    "zone_hint": "panel.example.com"
}

op is present (add the TXT record) or cleanup (remove it); several values may coexist at one name, so remove only the one in value. zone_hint is the host being validated; the plugin finds the provider zone that contains it. The plugin does the provider call with its approved http capability and ${secret:NAME} slots, and ends with acme_dns_result(1, ...) for done or acme_dns_result(0, message) for a refusal. A call that never answers, traps or runs past the call limit fails that order, and the runner still sends cleanup. The message is scrubbed of every secret the call expanded and shows in the site’s last_error.

The plugin never sees an ACME account or certificate key. The trigger is additive: modules that do not declare it are unaffected, so the ABI minor does not change. Calls go through the same bounded hook queue as webhooks, so a slow provider fails one issuance and never builds a backlog.

Approval

PluginHost::load(bytes, approved) takes the capabilities the operator approved for the module’s sha256. A module whose own declaration asks for more (a new trigger, log, state, or a bigger state cap) fails with CapsNotApproved, so a module cannot grant itself anything. It then runs with its own declaration, which is never wider than the approved set.

Limits

Every call runs in a fresh Store with a memory cap, an epoch deadline (Limits::call_timeout), a table-element cap (Limits::max_table_elements, checked at instantiation and on growth) and the log and state-write limits above. Modules are capped at 8 MiB. A call returns its Effects (state writes, log lines) and applies nothing itself: the caller commits them as one entry, which is how the controller will tag them with the leader term (automation hooks spec, “Commit rule”).

Compile bounds and the compile pool

Before a module is compiled, bounds::check walks it with wasmparser and rejects structure out of proportion to a real plugin (ModuleError::TooComplex, naming the bound): functions, types, locals per function (1000) and in total (50 000), block nesting depth, declared table (at most Limits::max_table_elements, 10 000) and memory sizes, globals, imports, exports and total code bytes (Bounds, set through Limits::bounds). CompilePool runs PluginHost::load on a fixed set of worker threads behind a bounded queue: a full queue answers Busy at once and a slow compile answers TimedOut (the compile itself cannot be interrupted, so it keeps its worker until done). The controller must load modules through the pool, never on its main loop. The workspace sets panic = "abort" for the release profile, so a panic in a compile worker (for example a wasmtime bug) ends the process rather than silently shrinking the pool; the bounds exist to keep hostile input away from that path.

A plugin’s state holds at most 1024 keys (a state_put of a new key past that returns -1). A declared state.max_bytes above 1 MiB is rejected, and PluginHost::new refuses a zero call_timeout.

Checking a built plugin

wayhouse-plugin-check <module.wasm> (crate wayhouse-plugin-host) loads the module with its own declaration as the approved set, calls init and, if declared, on_timer once, and prints what the host saw, including which capabilities were used. It exits 1 when the host would reject the module, a call fails, or the plugin used a capability it did not declare. Plugin repo CI is meant to run it on every build.

Installing a plugin (controller API)

Start a controller with --plugins (off by default; standalone or with --ha-peers). With --role slave, or without the flag, /plugins answers 501 with the reason. The routes sit behind the same admin bearer token as /config; X-Actor is recorded as created_by.

An install has a random id (route ownership will be plugin:<id>), a name (a-z, 0-9, -, up to 64 characters), the approved capability set bound to the module sha256, and a non-secret config (up to 64 KiB) that will be handed to init. Installs and blobs live in the controller’s sled database; a standalone controller keeps them there, an HA tier keeps them in the Raft state machine’s database, replicated (the blobs travel out of band, see “High availability”).

Ticks (standalone controller)

A scan runs once a second. An enabled install whose approved declaration has on_timer is called every tick_interval_secs; the first tick comes one full interval after the controller (or the enable) first sees the install, so enabling never runs a plugin inside the request. The module is compiled on first use through the bounded pool, init runs with the install’s config each time the module is loaded (first tick after a controller restart, after disable then enable, and after a failed load) against the state already stored, so it must tolerate existing state, and every call runs on the pool: a busy or slow pool skips the tick (shown in the status) and never blocks the controller; a skipped tick is retried at the next scan, not a whole interval later. One call per install is in flight at a time.

A call returns its state writes; the controller applies them as one batch only if the install’s state revision is still the one the call read (compare-and-set), then bumps the revision. A call that failed (trap, timeout, over a limit) commits nothing, is recorded in the status, and the plugin is called again at its next interval. Deleting an install removes its state; disabling keeps it, and enabling again resumes against it. created_by is a label taken from the X-Actor header the caller sent, not a verified identity (the admin token is what authorises the call). This is the single-node form of the design’s term-checked commit; the HA form is below.

The Plugins page (web UI)

wayhouse-ui proxies the controller’s /plugins API (/api/plugins*, needs --controller-url). Listing is viewer-level; upload, install, enable, disable and delete need the admin role. The page lists installs with their approved capabilities in plain words. “Upload plugin” sends the module (up to 8 MiB) to the controller, shows what it declares and installs it only after the operator ticks the approval box; the UI approves exactly the declared set, never an edited one. A controller that does not serve plugins (off, slave) shows its reason instead of a list. Not in the page yet: registry install, secrets, config editing.

High availability

With --ha-peers every replica holds the same installs and the same per-install state, in the log and in snapshots (module blobs are in neither; see “Module bytes”). The entries are an install, enable or disable, delete (state goes with it), and one state commit per plugin call. Applying is deterministic and a refusal is a normal answer, never a storage error: an install whose id exists, a malformed install (id, name, sha256, size, created_by, config size), a state commit over the host’s bounds, a stale revision, or a commit tagged with another term than the one the entry was appended in changes nothing, on every replica alike. The plugin trees record the highest Raft index applied, in the same transaction as the entry and restored from a snapshot, so a replay after a crash or a snapshot install changes nothing.

The API. GET /plugins and GET /plugins/{id} answer from the replica that got the request. Enable, disable and delete are proposed through Raft, and a follower forwards them to the leader like the other write routes. An upload, an install and a status read run on the leader (a follower forwards the whole request): the leader needs the module bytes to compile and check the approval before it proposes the install, and only the leader has tick results. The leader checks the record before proposing and every replica checks it again when applying.

The tick. Only the Raft leader ticks. A node that is not the leader drops its schedule, its loaded plugins and its statuses; when it wins an election every install starts over with a full interval of grace, so a new leader has applied the earlier terms’ entries before it runs a plugin. A tick’s state writes are one PluginState entry tagged with the term the leader read the state in and the state revision the call read: if leadership moved or another commit got in first, the state machine drops it and the status says so. A commit is refused before it reaches the log when over the host’s state limits (1 MiB, 1024 keys, 256 writes per call). Status is in memory on the leader and starts empty after a failover.

Module bytes (slice 7). The Raft log carries install records only; the module (up to 8 MiB) is a content-addressed blob each replica keeps in its own store and moves over the peer channel (--ha-token, the protocol gate):

• The leader pushes a module (PUT /raft/plugin-blob/{sha256}, verified against its hash on arrival) to every member and records the install only once a quorum of voters holds it, so an install cannot be stranded on a node that dies right after the upload. A quorum that cannot be reached answers 503.
• Every replica fetches the modules its installs reference and it lacks (GET /raft/plugin-blob/{sha256}, leader first, then the others, verified before it is kept) every few seconds: a replica that was down, joined late or restored a snapshot catches up this way. Until it has the module a node runs nothing for that install; a leader that cannot find one logs a repeated warning, since it would otherwise silently run nothing.
• Rolling upgrades: a new install is refused with 503 unless every member answers /raft/whoami with the current plugin_support level (3; a build before module replication leaves it out). A member started without --plugins, or as a slave, reports 0, so every replica needs --plugins before plugins are installed or their webhooks, secrets or key rotation change. A leader without --plugins would otherwise run no ticks and fail webhook forwards silently.
• The same gate covers secret changes, so setting, deleting or rewrapping a secret answers 409 while any cluster member (voter or learner) is unreachable. Revoking one credential therefore needs every member up; disabling or deleting the install still works as emergency containment. Existing installs keep working through an upgrade. Upgrade every replica before the first install; the gate does not cover a replica added later with an old build.
• An install that lands on a leader that has meanwhile lost leadership answers 503 retry and is never forwarded for a second hop.
• A controller removes the module blobs no install references (uploads that were never installed, and cleanups a crash interrupted) once at startup, so upload and install belong in one sitting. Under HA the sweep waits until the node knows a leader and has applied its whole log, because a blob pushed for a committed install that is not replayed yet looks unused. Uploads no install uses are capped at 64 MiB in total (507), so an open controller cannot be filled with them.
• A delete that lands between an install’s blob check and its commit removes the module on every replica; the leader puts it back and pushes it again once the install is recorded.