Operate Web Fleet

Clustering

Run Web Fleet's fleet configuration as an explicitly replicated SQLite cluster using the shared Gantry Core Raft runtime.

Initial clustered surface

The first clustered surface is intentionally narrow: groups, sites and site tags are consensus-owned, along with stable cluster identities and durable replication operation state. Numeric SQLite row IDs remain local materialization details.

Monitoring/check history, incidents, audit and crawl observations, TLS/DNS/performance data, analytics, deployments, notification/webhook state, users/sessions/API tokens, launcher state and external side effects remain node-local in this first release. Raft replay never runs checks, crawls, notifications or webhooks.

Note: materializing a site also creates its default monitor, health row and security-header expectations on every replica as part of the consensus-owned site definition — those are replicated, not node-local. Anything those defaults subsequently observe (check results, history, incidents) stays node-local.

The older propagation subsystem remains separate from Raft replication. While clustering is enabled, propagation of site/request definitions is rejected so it cannot bypass consensus-owned fleet topology; monitor-definition propagation remains node-local and does not make monitor execution consensus-owned.

Authority and failure behaviour

standalone                 -> ordinary local fleet writes
replicated ready leader    -> Raft proposal
replicated ready follower  -> authenticated forward to leader
replicated unavailable     -> reject; never fall back to local SQL

A follower waits until the committed entry is locally applied before returning success. Idempotency-Key is carried as the durable operation identity so a retry cannot silently become a second fleet mutation.

Storage boundary

Clustered materialization is SQLite-only in the initial implementation. Enabling replication with PostgreSQL fails closed. A node must start clustering with a clean site/group configuration; automatic merging of independently populated Web Fleet databases is deliberately not supported.

Configuration

WEBFLEET_REPLICATION_ENABLED=true
WEBFLEET_REPLICATION_NODE_ID=wf-node-a
WEBFLEET_REPLICATION_LISTEN=10.0.0.10:7440
WEBFLEET_REPLICATION_BOOTSTRAP=true
WEBFLEET_REPLICATION_TLS_CERT=/etc/webfleet/replication.crt
WEBFLEET_REPLICATION_TLS_KEY=/etc/webfleet/replication.key
WEBFLEET_REPLICATION_TLS_CA=/etc/webfleet/replication-ca.crt

Production replication requires the complete cert/key/CA mTLS configuration. Plaintext is an explicit development/test opt-in only.

Pairing and voter membership

Gantry pairing and Raft membership are separate. Pair every replication participant using Web Fleet's cluster trust flow, then add voters explicitly. Pairing a node never silently grants it a consensus vote. webfleet replicate join rejects the request unless the target is an active, paired Gantry member that advertises the replication capability and a compatible protocol — an unpaired or disabled peer is refused before the Raft configuration changes, so an invalid join cannot make the cluster read-only.

webfleet replicate status --user admin@example.com --password '...'
webfleet replicate join --user admin@example.com --password '...' wf-node-b 10.0.0.11:7440
webfleet replicate snapshot --user admin@example.com --password '...'

The replication operator endpoints use the normal Web Fleet administrator session/CSRF boundary.

Quorum

Raft owns quorum. Three voters require two for authoritative writes; losing quorum makes fleet mutations fail closed. The implementation does not contain a three-node ceiling, and Web Fleet itself is locally certified at three voters (with a five-voter smoke certification also passing) through the same generic path used by the shared Gantry runtime. Real-infrastructure dogfood has not been performed yet, so no arbitrary-size production claims are made before that evidence exists.

Snapshots and recovery

Snapshots carry the consensus-owned group/site/tag state and durable operation identities. Restore reconciles existing clustered rows in place, preserving their local numeric IDs so node-local monitoring history is not destroyed merely by consensus recovery. Site deletion remains a real fleet deletion and therefore follows the normal dependent-data semantics.

Security model

Raft transport uses mTLS plus Gantry's pairwise peer credentials, identity binding, capability checks, replay protection and peer revalidation. Disabling or revoking a peer removes its replication authority; Raft voter membership remains a separate operator decision. The replicated topology payloads (group, site, tag definitions) contain no credentials; site URLs are validated to reject embedded credentials, so consensus-owned site definitions remain credential-free. Every voter stores the consensus-owned topology — site addresses, group structure and tags — so every voter is a trusted holder of that replicated fleet information. There is no encryption at rest.

Monitoring ownership and replay isolation

Clustering replicates topology, not monitoring execution. Each node independently runs its own local scheduler against the replicated fleet: new, updated and deleted sites are reconciled by each node's ordinary local monitoring machinery, and monitoring observations/results/history are node-local and may legitimately differ between nodes while topology converges. Raft replay, snapshot restore, trailing-log replay and follower catch-up only materialize topology via pure SQL; they never perform HTTP checks, DNS queries, TLS probes, crawls, notifications or webhooks. Those external effects happen only when a node's own scheduler later processes the recovered topology through its normal path.

Node identity and deployment contract

WEBFLEET_REPLICATION_NODE_ID is optional. When unset, the Raft server identity is derived automatically from the durable Gantry/Webfleet cluster identity (the wf_... node id created at startup); this is the recommended production path. If it is set, it becomes the Raft server id and must match the identity node id used by webfleet replicate join. The durable node id, the human-readable display name, the public endpoint (https://IP-or-hostname) and the Raft address (IP:7440) are distinct; do not use the display name as the Raft identity. Webfleet has no default replication port — WEBFLEET_REPLICATION_LISTEN is required — and a common port across independent hosts is the normal deployment pattern.

Request limits and join preflight

The cluster/proposal transport enforces Gantry's MaxRequestBytes limit, and the application topology API rejects oversized or unknown-field requests. webfleet replicate join rejects the request unless the target is an active, paired, replication-capable, compatible Gantry member, so an invalid join cannot make the cluster read-only; Core's capability checks remain defense in depth.

Evidence levels and deferred walls

Webfleet clustering is locally certified at three voters (production-shaped certification) and a five-voter smoke certification, through the generic Gantry path; the architecture supports arbitrary voter counts. Real-infrastructure walls remain for fresh hosts: a true two-channel network partition, scheduler-active behavior under real networking, mTLS deployment mistakes, resource/log observations and deliberately divergent monitoring histories across independent hosts.