Upgrading a fleet

How to move a running deployment to a newer version one component at a time, without a flag day. The design and its reasoning are in the component versioning spec; this page is the operator’s view of what is built. Releases and tags are in RELEASING.md.

Compatibility rules

• Window: N and N-1. Two adjacent minor versions of the product (0.N and 0.(N-1)) are meant to run side by side during an upgrade. While the product is 0.x, “minor” is the second number.
• Protocol major must match. A node that receives another major (or an unreadable X-Wayhouse-Protocol header) answers 426 and counts it in wayhouse_protocol_mismatch_total. A breaking wire change bumps the major and, to keep the window honest, must ship with a gate that also admits the previous major for one product minor and routes those requests to a compatibility shim; every other major still gets 426. Today the gate admits only the current major and no shim exists, because no second major exists yet.
• Protocol minors are additive. A sender only uses a newer optional field or route with a peer that has said it understands it: every request carries the caller’s version and every response the server’s, and a peer whose version is not known yet gets the baseline. Receivers ignore unknown JSON fields.
• Config is the exception to “controllers first”: the config schema_version is the minimum schema a document needs, and the controller refuses (422) a document newer than the lowest max_config_schema any live proxy registration reported. Upgrade every proxy before you use a config field the new version introduced.
• Store: a controller refuses to open a store written by a newer build and migrates an older one once, on open.

What is and is not checked: the header check enforces protocol majors only. The N / N-1 product window itself is a rule kept by review (see the AGENTS.md row for wire changes) and exercised by tests that inject an older protocol minor into a real fleet (crates/wayhouse-fleet-tests/tests/mixed_versions.rs). Once two releases exist, the non-required compat workflow also runs the previous release’s images against the current ones.

Version table

One row per release, newest first. A CI check (.github/scripts/check_upgrading_doc.py) fails when the first row disagrees with the constants in the source, so a protocol, schema, store or ABI change cannot land without its row. Use unreleased for the product column until the release is tagged.

Upgrading from 0.1.0 to 0.2.0

0.2.0 is the first release with a real upgrade step. Things to know before you start:

• Gossip does not cross the version. 0.2.0 gossip frames carry a magic byte and GOSSIP_VERSION; a 0.1.0 node and a 0.2.0 node drop each other’s datagrams. While a fleet runs both, health votes and domain quorum are not shared between old and new nodes, and each node falls back to its own checks. Existing player traffic is not affected, but nodes can briefly disagree on backend health, so new sessions may be placed differently on old and new nodes. This is the one exception to the N and N-1 window (see docs/10).
• Registry names. A name must be 1 to 64 characters of A-Za-z0-9._- and not subscribe; see “Dedupe first” below for the check to run before the first replica is upgraded.
• Bundled sniffers moved. The official WASM sniffers now live in wayhouse-proxy/sniffers and are no longer built from this repository. Get them from that repository’s releases, install them from the UI’s registry page, or copy them into the directory named by settings.sniffers.dir (see sniffers.md). sniffers.lock in this repository only pins the releases the CI loader tests run against; it has no effect on a deployment.
• Config schema 2. listeners.plugin_routes is a schema 2 field. A 0.1.0 proxy cannot parse it, so use it only after every proxy runs 0.2.0. The controller enforces this on submit (422 naming the lowest live proxy maximum). A proxy that does not announce a schema (0.1.0) is sent only revisions that need schema 1: one that first starts after the first schema 2 revision exists gets the newest schema 1 revision, not the newer ones, and keeps it until it is upgraded.
• Plugin ABI. The version table has no plugin column: the plugin ABI is 0.1 and the host requires an exact minor match while the major is 0.

Order of operations

1. Aggregator and UI (the read side).
2. Controller tiers, leaf slave tiers first and the root tier last. A slave tier is a child controller under a parent. Inside the window either order works because of the gating rule above; going leaves-first keeps the root, which owns writes, on the version your operators have run longest.
3. Raft HA inside one tier, see below. A tier made of several controllers is a different mechanism from the parent and child tiers.
4. Proxies and agents, a few at a time: drain, upgrade, rejoin.
5. Config fields introduced by the new version, only after every proxy runs it.

Check the fleet view in the UI between steps: an instance more than one minor behind the newest, on another protocol major, or that refused a request for an incompatible protocol in the last five minutes is red; one minor behind is yellow.

The fleet view lists only the instances that push to the aggregator, which are the proxies. Controllers, the aggregator, the UI and agents do not appear in it. For those, read wayhouse_build_info on each component’s /metrics between steps (its version and protocol labels).

Controller tiers

Upgrade a tier completely (all its replicas, see below) before moving to its parent. Parent-to-child and child-to-parent calls both carry protocol versions, so a child one minor older than its parent is sent only baseline fields. A slave tier relays writes upward through its leader; a tier mid-upgrade keeps relaying as long as the protocol major matches.

The baseline-fields rule for a child one minor older is not covered by an automated test for the tier path (the fleet tests exercise a proxy against a controller, not a slave tier against its parent), and neither is the relay cursor across versions (a child resuming the parent’s revision stream after one side upgraded). Treat the first tier upgrade in your environment as a rehearsal, and read the “N and N-1” window above as tested for proxy against controller only.

Raft HA inside a tier

Replicas of one tier replicate through Raft; /raft/* carries the same protocol header check, and changing a raft RPC payload counts as a breaking wire change.

1. Upgrade the followers one at a time. Wait for the upgraded node to catch up (GET /admin/ha/members shows it as a voter again) before touching the next.
2. There is no leader-transfer command. To move leadership, stop the leader process; the followers elect a new leader within the election timeout and writes resume through it.
3. Upgrade the former leader last.

Dedupe first. Since 0.2.0 a tier’s replicated registries refuse a registration whose WireGuard public key another name already holds (409). That check runs when a log entry is applied, so a 0.2.0 node can reject an entry that a 0.1.0 node accepted, and a fresh node replaying an old log does the same; the replicas would then hold different registry contents. Before the first replica is upgraded, list GET /peers and GET /proxy-peers on the tier and DELETE every extra name that shares a public key with another, keeping the one that is in use. Also rename every name outside [A-Za-z0-9._-], longer than 64 bytes, or equal to subscribe: since 0.2.0 a new registration with such a name is refused at submit time (422). Entries already in the log are not revalidated when applied, but a node whose registration refresh is refused under 0.2.0 drops out of the registry.

This sequence is not covered by an automated test (the fleet tests do not run a rolling Raft upgrade); treat the first use in your environment as a rehearsal. A write that a follower forwards to the leader is sent with the follower’s protocol version, not the original caller’s; that is harmless inside the window because receivers ignore fields they do not know.

Proxies

A proxy upgrade relies on draining:

1. POST /admin/drain: the instance reports not ready, so the load balancer or anycast routes new clients elsewhere. The data path keeps running.
2. Wait for sessions to end (active_conns in GET /config, or let shutdown_grace_sec expire).
3. Stop the old process, start the new one, check GET /readyz, and let the load balancer bring it back.

A config reload that replaces a UDP listener no longer drops new flows while the old group drains (#186), but it ends that listener’s live sessions: the old workers close their sockets at once, and each client opens a fresh session in the replacement group on its next datagram (possibly from a new upstream source port). Plan a reload that changes a UDP listener like a short blip for its players. A listener that is removed, or whose bind address changed, still drains as described above. See A config reload that replaces a UDP listener.

Agents

Agents register with the controller like proxies do. Upgrade them after the controller tier they register with; an agent one minor older than its controller works against baseline fields.

Kubernetes

The same order applies, with a rolling update (maxUnavailable: 1) and the readiness probe gating each step. Pending #88: the packaged manifests for this do not exist yet, so this section describes the intent, not a tested procedure.

Rollback

Roll back in the reverse order. A downgrade is safe inside the window for the protocol. A controller store that a newer build has migrated cannot be opened by an older build (FormatTooNew): restore the store from a backup taken before the upgrade, or keep the upgraded controller. Config documents that use a field of the newer schema are rejected by older nodes, so revert the controller to the previous config revision first.

Troubleshooting