Aeron® Cluster Placement: Choosing an AZ Topology

Where you place the nodes of a Raft cluster is a single decision that sets both your hot-path latency and your worst-case blast radius — and the two pull in opposite directions. This page lays out the base topologies for a matching-engine (ME) cluster — three symmetric bets and two hybrids between them — scores them, and names the middle ground most exchanges actually ship.

For the byte-level consensus mechanics — terms, log replication, commit — see The Aeron Files. This page is the placement decision: which topology, why, and what each one costs you on p50/p99, availability, and the operations budget.

The one trade-off behind every choice

The hot path is OMS → Raft leader (the order-write path). Only the leader accepts writes, so what the exchange actually feels is the latency from the order-entry point to wherever the leader sits. Every placement decision is answering one question:

How much hot-path latency will you spend to shrink the blast radius of a failure?

Keep the leader close and you keep p50/p99 tight; spread the cluster for resilience and you drag part of that hot path — or the commit quorum — across an availability-zone boundary. There is no placement that wins both. The rest of this page is about picking your point on that curve deliberately.

Scoring dimensions

Seven dimensions, higher is better:

Fault tolerance — the largest failure domain you survive: node → rack → datacenter → AZ → region.
Hot-path latency — OMS↔leader latency. Intra-rack µs, or cross-AZ ms? Higher score = lower latency.
Availability — does the cluster ride out a failure automatically and keep serving?
Recovery automation — automatic re-election, vs a manual DR cutover, vs a cold rebuild.
Data safety (RPO) — risk of losing committed / in-flight messages at a switchover.
Cost efficiency — cross-AZ traffic, node count, bandwidth. Higher score = cheaper.
Operational simplicity — day-to-day burden: leader pinning, replication monitoring, DR runbooks.

T1 — single cluster, multi-AZ

Three (or five) Raft nodes, one per AZ. The textbook HA layout: one consensus group stretched across the region.

The cost: after any election the leader can land in AZ-2 or AZ-3. The moment it drifts, every order pays a cross-AZ round trip on the OMS→leader hop until you move it back. And even with the leader pinned local, commit still needs a cross-AZ majority ack — voting members are spread across AZs, so at least one cross-AZ ack sits on the commit path regardless of where the leader sits. Pinning removes cross-AZ latency from the client-facing path, not from commit.

That leader-drift problem is not fatal — it is exactly what the preferred-leader pattern solves. See Leader Placement and Preferred-Leader Control for biasing elections toward a chosen AZ and reclaiming the leader after a failover, and Tuning Graceful Step-Down Time for how fast that reclaim is (measured ~57 ms tuned, ~266 ms stock).

Dimension	Score
Fault tolerance	5 — survives a whole-AZ loss
Hot-path latency	2 — leader drift + always-cross-AZ commit
Availability	5 — automatic re-election, no human in the loop
Recovery automation	5
Data safety (RPO)	5 — zero loss, synchronous majority
Cost efficiency	3 — same node count as T3 but you pay cross-AZ traffic continuously
Operational simplicity	2 — leader pinning + step-down is the heaviest to run
Total	27 / 35

Pick T1 when a regulator mandates availability, an AZ loss must fail over automatically with zero RPO, and cross-AZ hot-path latency is acceptable.

T1.5 — stretched 2+1 (local majority)

One 3-node Raft cluster, split 2+1: two voters share one AZ, the third sits in a second AZ. The trick is quorum math — a majority of 3 is 2, so the leader and its co-located peer form a quorum on their own. Commit is acked intra-AZ, in µs; the AZ-2 voter is a full member but never sits on the commit path as long as both hot-AZ voters are up and the leader is pinned local.

That buys T3-class commit latency from a single stretched voting cluster — something neither T1 (always a cross-AZ commit ack) nor T2.5 (its cross-AZ node doesn’t vote) delivers. The price is asymmetric fault tolerance, and it is the whole story here:

Lose the minority AZ (the lone AZ-2 voter) or any single node → the cluster rides it out automatically. The remaining two voters are still a majority, commit stays intra-AZ, and RPO is zero — that node was never on the commit path.
Lose the majority AZ (both hot-AZ voters) → the cluster halts. The lone survivor can’t reach quorum (1 < 2), so recovery is a manual single-node reconfigure with RPO > 0 — the AZ-2 voter can lag, because commits never waited for it. “Multi-AZ” here does not mean “auto-survives an AZ loss” the way T1 does.
Leader pinning is mandatory. If an election lands the leader on the AZ-2 node, commit now needs a hot-AZ ack → cross-AZ commit, and the latency win evaporates. Pin the preferred leader to the majority AZ — same tool as T1’s leader-drift fix, see Leader Placement and Preferred-Leader Control.
Only two voters in the hot AZ. Lose one and commit quorum immediately needs the cross-AZ vote, so p99 degrades until you repair. This is exactly what T2.5’s extra node buys back — T2.5 keeps three voters in the hot AZ and rides a hot-AZ node loss with no cross-AZ penalty.
Placement-group choice couples or isolates the two hot voters. A cluster placement group packs them for the tightest intra-AZ ack, but on shared infrastructure — a single rack/CPG fault takes both at once and halts the cluster. A spread placement group within the AZ gives the two voters independent failure domains for a few µs more per ack. On a matching engine, usually take the spread.

Dimension	Score
Fault tolerance	3 — auto-survives a node or the minority AZ; the majority AZ halts (manual)
Hot-path latency	4 — intra-AZ commit while pinned, but only two hot-AZ voters; drops to cross-AZ if one dies
Availability	4 — rides node loss and minority-AZ loss; majority-AZ loss stops the cluster
Recovery automation	3 — node-level auto-elect; majority-AZ loss is a manual single-node reconfigure
Data safety (RPO)	3 — zero on the common minority-AZ loss, > 0 if the majority AZ dies
Cost efficiency	4 — one cluster, three nodes; one cross-AZ log stream, not a per-commit cross-AZ ack
Operational simplicity	3 — mandatory leader pinning + an asymmetric-failure runbook, but a single 3-node cluster
Total	24 / 35

Pick T1.5 when you have only two AZs (can’t do a symmetric 1-1-1 spread), want the cheapest single stretched cluster with intra-AZ commit latency, and can accept that only the minority AZ loss auto-recovers — the majority AZ dying is a manual, RPO > 0 event. With three AZs, T1 or T2.5 usually dominate it.

T2 — primary + standby (two clusters)

Two independent Aeron® clusters. The primary serves live traffic in one AZ; its archived log replicates asynchronously to a standby cluster in another AZ. A real disaster is a manual operator cutover.

Because replication is async and off the hot path, the primary runs at single-AZ speed — p50/p99 identical to T3. The catch is on the standby: it lags, so a hard cutover can drop the messages that hadn’t replicated yet (RPO > 0).

Two ways to fan the log across the AZ boundary, both async:

Relay / daisy-chain (1 cross-AZ stream). The leader ships to one standby node, which forwards to the other two intra-AZ. Cheapest cross-AZ bandwidth; one extra intra-AZ hop, so slightly higher standby tail lag.
Broadcast (3 cross-AZ streams). The leader ships to each standby node directly. Lowest standby lag and lowest cutover RPO, at 3× the cross-AZ bandwidth.

Relay / daisy-chain — one WAN stream, then intra-AZ fan-out:

Broadcast — the leader ships to every standby directly:

Dimension	Score
Fault tolerance	4
Hot-path latency	5 — serving cluster is single-AZ, same as T3
Availability	4 — node failures behave exactly like T3 (auto-elect, zero loss)
Recovery automation	4 — AZ-level cutover is manual: detect, promote, repoint
Data safety (RPO)	4 — > 0 only on a hard cutover
Cost efficiency	1 — 6 nodes (3 idle standby) + replication bandwidth; the most expensive
Operational simplicity	2 — two clusters to watch, replication-lag monitoring, DR drills
Total	24 / 35

Pick T2 when you want intra-AZ hot-path latency and survival of a full-AZ loss, and can accept a few-message RPO plus a longer manual RTO on the rare hard cutover. This is the classic CEX balance point. The mechanics — warm standby, in-place identity switch, background snapshots, daisy-chaining — are covered in Cluster Standby and Multi-AZ HA Design.

T3 — single cluster, single AZ

Every Raft node in one AZ. The lowest-latency layout, for when the hot path is non-negotiable and AZ-level DR is handled elsewhere (often a T2 layer on top).

single-AZ is not single-rack. Spread the nodes across datacenters within the AZ (spread placement groups / a TAM-coordinated layout) to keep intra-AZ latency while getting real fault isolation between nodes.

Dimension	Score
Fault tolerance	2 — no AZ-level tolerance; the AZ falls, the cluster stops
Hot-path latency	5 — intra-AZ / intra-rack µs, no leader drift, all nodes equidistant to OMS
Availability	3
Recovery automation	4
Data safety (RPO)	3 — worst in a disaster: every replica sits in the AZ that died
Cost efficiency	5 — one cluster, no replication
Operational simplicity	5 — nothing to pin, nothing to replicate
Total	27 / 35

Pick T3 when matching must squeeze every microsecond and AZ-level DR is solved separately.

T2.5 — single-AZ cluster + one cross-AZ standby node (the value pick)

The middle ground most latency-sensitive exchanges actually ship: run T3 for the hot path, then attach one cross-AZ Cluster Standby node that continuously pulls log + snapshots into another AZ.

You keep T3’s µs hot path and its low cost (one extra node, not three), and buy a cross-AZ safety net: a disaster no longer loses everything, because a warm, caught-up copy of the log lives in another AZ. The standby applies no back-pressure on the leader — replication is async — so p50/p99/throughput on the primary are unchanged. It is the cheapest way to move RPO off “whatever the dead AZ still held.”

The trade vs T2: you have one standby node, not a full three-node standby cluster, so promoting it into service after an AZ loss is more hands-on and slower than T2’s cluster-to-cluster cutover. If a few-minute, more-manual RTO on a rare event is fine, T2.5 gives you ~90% of T2’s protection at a fraction of the cost.

Dimension	Score
Fault tolerance	3 — a warm cross-AZ copy survives, but promotion is manual
Hot-path latency	5 — identical to T3; the standby is async and applies no back-pressure
Availability	3 — node loss auto-elects (it’s a live T3 cluster); an AZ loss needs a manual promote
Recovery automation	3 — within-AZ auto; the cross-AZ promote is one hands-on node, slower than T2’s cutover
Data safety (RPO)	4 — a caught-up copy lives off-AZ; > 0 only on a hard AZ loss
Cost efficiency	4 — one cluster + one extra node and one log stream; far cheaper than T2’s idle trio
Operational simplicity	4 — one cluster to run plus replication-lag monitoring and a single-node promote runbook
Total	26 / 35

Comparison matrix

Dimension (higher = better)	T1 · multi-AZ	T1.5 · 2+1	T2 · primary+standby	T2.5 · +standby node	T3 · single-AZ
Fault tolerance	5	3	4	3	2
Hot-path latency	2	4	5	5	5
Availability	5	4	4	3	3
Recovery automation	5	3	4	3	4
Data safety (RPO)	5	3	4	4	3
Cost efficiency	3	4	1	4	5
Operational simplicity	2	3	2	4	5
Total / 35	27	24	24	26	27

The two hybrids read as the diagonal of this matrix. T1.5 is a single stretched voting cluster biased toward latency: it co-locates the quorum to buy intra-AZ commit (HPL 4 vs T1’s 2), paying for it in asymmetric fault tolerance (3 — only the minority AZ loss auto-recovers). T2.5 is T3 (same hot-path numbers) with fault tolerance/RPO nudged up and one extra node — a lean toward the T3 corner with an async cross-AZ hedge. Neither maxes any single column; both exist to dodge the corners’ worst weaknesses.

RPO and RTO by failure mode

The “Data safety (RPO)” score above collapses two numbers a DR plan keeps separate: RPO — how much committed data a failure can lose — and RTO — how long you’re down before service resumes. Both depend entirely on which failure domain dies, so read them per failure mode, not as one figure. RTO “auto” means Raft re-elects with no human in the loop (tune it in Tuning Cluster Failover Time); “manual” means an operator runbook — promote/repoint — measured in minutes.

Topology	Single-node loss	Full-AZ loss
T1 · multi-AZ	RPO 0 · RTO auto	RPO 0 · RTO auto — the only topology that auto-survives an AZ loss at zero data loss
T1.5 · 2+1	RPO 0 · RTO auto	Minority AZ: RPO 0 · RTO auto. Majority AZ: RPO > 0 (lagging voter) · RTO manual — single-node reconfigure
T2 · primary + standby	RPO 0 · RTO auto (within the primary)	RPO > 0 (async replication lag) · RTO manual — cluster-to-cluster cutover
T2.5 · +standby node	RPO 0 · RTO auto (within the primary)	RPO > 0 (async replication lag) · RTO manual — single-node promote, more hands-on/slower than T2
T3 · single-AZ	RPO 0 · RTO auto	RPO = every in-AZ replica (no DR at this layer) · RTO = external DR / rebuild, unless a T2 layer sits on top

The pattern: RTO is automatic for any failure the cluster still holds a quorum through, and manual the moment recovery crosses an AZ boundary that the commit path never spanned. RPO is zero exactly where commit was synchronous (T1 always; T1.5/T2/T2.5/T3 for a node loss) and positive wherever the surviving copy was fed asynchronously (T1.5’s majority-AZ loss, T2/T2.5’s AZ cutover) or didn’t exist (T3’s AZ loss).

How to choose

Walk the tree top-down — each fork is one question about your SLA, and the leaf is the topology to start from. The table below is the same decision as a lookup, with the reasoning spelled out.

Your priority	Topology
Availability ≫ latency — regulated, must auto-survive an AZ loss at zero RPO, cross-AZ hot-path latency acceptable	T1
Intra-AZ latency from one cluster, only two AZs — µs commit from a single stretched cluster; fine that only the minority-AZ loss auto-recovers	T1.5 (stretched 2+1)
Latency and availability both — intra-AZ hot path and auto-survive a full-AZ loss; the common CEX balance	T2
Near-T3 performance + a cross-AZ hedge — squeeze latency but don’t want a disaster to lose everything	T2.5 (T3 + one cross-AZ standby node)
Pure performance — every microsecond matters, AZ-level DR handled separately	T3

Placement is a decision, not a lock-in

Two follow-on problems turn placement from a one-time bet into something you operate:

T1’s leader drift is fixable. Bias elections toward your chosen AZ and reclaim the leader after a failover — Leader Placement and Preferred-Leader Control — and tune how fast that reclaim runs in Tuning Graceful Step-Down Time.
Moving a live cluster between AZs or regions is a runbook, not a rebuild. When you outgrow the initial placement — or migrate to a new region — you can do it node-by-node (client-transparent, followers free) or as a whole-cluster swap, and the only meaningful downtime is the single leader crossover, which the graceful step-down knobs above shrink. See Migrating a Live Cluster.

See also: Cluster Standby and Multi-AZ HA Design, Aeron® Cluster and Raft Consensus, Tuning Cluster Failover Time (the unplanned-death budget).