Migrating a Live Aeron® Cluster Across Regions

Sooner or later you move a live cluster — a region migration, a new datacenter, a placement you’ve outgrown. The reassuring result: with Premium Cluster Standby this is a runbook, not a rebuild, and the only meaningful downtime is a single leader crossover you can shrink with the graceful step-down knobs. This page covers the two verified approaches, their measured cost, and the one hazard that silently breaks the transparent path.

It picks up where Aeron® Cluster Placement leaves off: that page is which topology; this one is how to move between them without a rebuild. All numbers below were measured on a single host (3-node cluster, no WAN) unless marked otherwise — treat them as the shape of the cost, not a WAN-latency prediction.

The setup: standbys already tailing in Region B

Both approaches start the same way, and both are pure Cluster Standby: the live 3-node cluster runs in Region A; three standby nodes in Region B continuously tail the log across the WAN. Standbys don’t vote, so consensus commits stay entirely local to A — no WAN in the hot path while you prepare. What differs is how you flip B from tailing to serving.

Two approaches

	Node-by-node (rolling)	Whole-cluster swap (cutover)
Mechanism	Replace members one at a time via PremiumClusterTool transition-as-member <id> + hostname repoint; same cluster throughout	Promote the whole B standby group (2-arg transition) into a new cluster; repoint clients
Cluster identity	One continuous cluster, members swapped in place	A new, separate cluster in B
Steps	N sequential swaps, DNS repoint + strict ordering per node	One promotion trigger + one client switch
Client endpoint change	No — hostnames reused, client auto-follows	Yes — new cluster, client must be repointed
Verified	Followers: yes — standby joins the live cluster in slot k, 3/3 healthy. Leader crossover: not cleanly converged on our single-host harness (see below)	Yes — group forms one cluster, client resumes

Step through both migrations below — toggle the scenario, then use Play or Next to watch each move. Watch the two meters: OMS → leader latency (the client-facing hop) and the consensus commit path (which crosses the WAN only when the voting majority sits away from the leader).

✔Node-by-node (rolling) ⇄Whole-cluster swap (cutover)

Node-by-node — 3 voting in A + 3 non-voting standbys in B, replaced one at a time

Region A

Region B

WAN ~40ms

Customers

external

gateway

Gateway A

live

Gateway B

live

oms

OMS A

active

OMS B

—

me cluster

A · node 0

leader

A · node 1

member

A · node 2

member

B · node 0

standby

B · node 1

standby

B · node 2

standby

OMS → ME leader latency

local ~0.1 mscross-region ~40 ms

LOCAL

ME consensus commit path

majority localmajority needs B (WAN)

LOCAL

0 / 5

‹ Prev ▶ Play Next ›

ME leader ME voting member ME standby (no vote) retired local flow cross-region (WAN) standby tailing log consensus membership consensus commit crossing WAN

Node-by-node — followers are free, the leader is the whole cost

Promote standbys into member slots one at a time. Replace the followers first, the leader last:

• Follower swaps cost ~0 client outage (measured: exactly 0 missing messages). The leader and quorum stay in A; a follower move is invisible to clients.
• One member at a time — Raft membership changes are single-server by design. Swapping two at once in a 3-node cluster momentarily drops below quorum and stalls. Pre-warmed standbys make each step fast (small tail replay, not a full snapshot pull).
• The single leader swap is the only client-visible hit. Removing the last A member — the leader — triggers an election a B member wins. Stock and ungraceful, that’s ~11.8 s (survivors wait out the ~10 s LEADER_HEARTBEAT_TIMEOUT, then a ~1 s election). The AeronCluster client auto-follows to the new leader on the pushed NewLeaderEvent — no app code, no endpoint change.

The leader crossover is the one step still carrying convergence risk

Follower swaps are clean and verified. The leader-slot swap is not: on our single-host harness, ungracefully stopping the leader and promoting the B node into its slot failed to re-converge — survivors rejected the promoted node’s higher-term vote (unexpected vote request, a term split) and the cluster stalled. Resolution itself succeeded (every survivor re-resolved the new leader IP), so this is a Raft election/term problem of the promote-into-slot leader step, not a DNS or resolver issue. Treat node-by-node’s leader crossover as carrying both the timing (~11.8 s) and a convergence risk until a graceful step-down / RESIGN / longer settle demonstrates a clean crossover. This is single-host only — WAN behavior is untested.

There is a transient window worth naming: once two of the three voting members live in B while the leader is still in A, the commit path crosses the WAN (the leader needs a B ack for its majority). The recommended sequence keeps that window short by moving straight to the leader step.

Whole-cluster swap — one bounded hit to everyone

Quiesce or fail the entire A cluster, promote all three B standbys as a group into a brand-new cluster, then repoint clients:

• One ~7–8 s outage, hitting all clients at once (measured, hard fail). No free follower step — the whole group moves together.
• The B group runs a fresh startup election (STOP_FOR_EXPORT → RECOVER_CONSENSUS_MODULE → CONSENSUS). There’s no incumbent leader to step down, so there is no graceful-handoff / RESIGN lever here — its ~7–8 s is close to a floor, not a reducible number.
• Clients must be repointed. B is a different cluster with different endpoints; the dead A cluster can’t advertise them. You give the client B’s ingress list explicitly (config/DNS push, or the switchToStandby pattern). Not client-transparent by construction.

Measured downtime

Phase	Node-by-node	Whole-cluster swap
Follower moves	~0 client outage (0 missing msgs)	n/a (whole group at once)
Leader transition	~11.8 s re-election (waits out LEADER_HEARTBEAT_TIMEOUT)	~7–8 s startup election of the new group
Transition itself	~2 s per node (off the client path for followers)	~0.4 s
Client failover once leader exists	<100 ms (auto-follow)	<100 ms (after switch)
Total client-visible outage	~11.8 s — but only at the single leader swap	~7–8 s, all clients at once
Client endpoint change	No	Yes

Read the totals carefully: node-by-node spreads the work over N steps of which only the leader swap is client-visible; whole-cluster swap is one hit to everyone. “Which is faster” reduces to how cheap you can make the one leader election — which is the next section.

Surprise finding: a graceful stop did NOT beat the timeout

On the tested shutdown path, killing the leader abruptly (docker kill -9) took ~11.6 s to a new leader serving; a clean docker stop (JVM exits in 0.24 s) took ~11.1 s — essentially the same. The fast end-of-stream election path did not fire from the shutdown hook, so followers still waited out the ~10 s heartbeat timeout. Do not assume a clean shutdown makes the leader swap sub-second — it doesn’t. Sub-second needs the RESIGN fork below.

Completing the story: making the leader crossover cheap

The leader crossover is the only reducible downtime in either approach, and it’s the same lever as routine leader rebalancing — so the placement pages already cover it. This is where migration and day-to-day operations converge:

• Lower LEADER_HEARTBEAT_TIMEOUT (default 10 s → 1–2 s) — the single dominant term of the leader blackout, config-only. The cost is more false-positive elections on a jittery/high-latency link. See Tuning Cluster Failover Time.
• Graceful step-down / RESIGN — an explicit voluntary resign forces the fast election path deterministically. Measured: a planned handoff is ~57 ms tuned / ~266 ms stock, versus the ~11 s an unplanned death costs — ~40× cheaper before tuning, ~190× after. This is the same mechanism you use to reclaim the leader’s AZ after any failover. Full walk-through: Leader Placement and Preferred-Leader Control and the knob-by-knob timeline in Tuning Graceful Step-Down Time.

With RESIGN, node-by-node’s leader step drops toward ~1–2 s / sub-second — below the whole-cluster swap floor — while its follower steps stay free. That inverts the stock ranking: with a graceful step-down, node-by-node is both faster and client-transparent.

The DNS / JVM-cache hazard (node-by-node, hostname reuse)

Node-by-node is only client-transparent if the B nodes reuse A’s hostnames and every JVM re-resolves the repointed name. This is load-bearing and easy to get wrong — but which resolver you run decides whether the hazard exists at all:

• Aeron re-resolves periodically (aeron.driver.reresolution.check.interval, default 1 s). The resolver is pluggable (MediaDriver.Context.nameResolver(...)); the exposure is entirely about which one you pick.
• With the default (InetAddress) resolver — hazard present. DefaultNameResolver calls InetAddress.getByName, which is subject to the JVM DNS cache (networkaddress.cache.ttl, cache-forever under a SecurityManager). If the JVM caches the old IP, Aeron’s 1 s re-resolution loop keeps getting the stale answer and the repoint silently no-ops — a member stuck on the dead IP with no error from Raft. Miss -Dnetworkaddress.cache.ttl=0 on any node or the client and you hit this. Mitigate with cache.ttl=0 everywhere and prefer an /etc/hosts rewrite (instant, local) over external DNS for a fast cutover. These reduce the hazard; they don’t remove it.
• With a custom NameResolver — hazard eliminated (verified). A resolver that returns addresses from your own control plane via InetAddress.getByAddress(name, rawBytes) never touches the JVM DNS cache or the platform resolver at all. An EC2 run proved this the hardest possible way: with the JVM cache pinned high (networkaddress.cache.ttl=600) and the Aeron identity names absent from DNS entirely, a node-by-node follower swap still joined the live cluster and the repoint took effect in ~1 s — because resolution went through the custom resolver, not InetAddress. A Consul/etcd/config-service/file-backed NameResolver is the right choice for a serious deployment.
• Either way, this does not gate the outage — the client-visible downtime is the leader election, which happens with zero DNS involvement. Resolution only governs when a replacement member becomes reachable (i.e. when 3/3 fault tolerance is restored).

Net for the decision: the DNS-cache hazard is real only if you rely on InetAddress/DNS//etc/hosts. If you can deploy a custom resolver, it stops being a reason to prefer cluster-swap. The whole-cluster swap sidesteps it structurally instead: B uses distinct endpoints, so the client connects fresh to new names — there’s no “same name, new IP” for a stale cache to defeat. (It only matters if you deliberately reuse A’s hostnames for B.)

When to choose which

Choose…	When…
Whole-cluster swap	You want the simpler path and can tolerate a bounded ~7–8 s hit to all clients at once. Default when you don’t have (and don’t want to maintain) the RESIGN fork. Client must be repointed to B’s endpoints.
Node-by-node	You need client-transparent member moves (hostname reuse, no endpoint change) with followers migrating at ~0 outage; and/or you can deploy the RESIGN fork (or a low heartbeat timeout) so the one leader crossover is sub-second. Costs: N-step complexity; the same-hostname discipline (plus either cache.ttl=0//etc/hosts or a custom NameResolver — the latter removes the DNS-cache hazard entirely); a reduced-fault-tolerance (2/3, no spare) window during each swap; and a leader crossover that is ~11.8 s ungraceful and whose clean convergence we haven’t yet demonstrated (followers are free and verified).

Bottom line

Whole-cluster swap is verified end-to-end; node-by-node’s follower swaps are verified, and its leader crossover is the one step still carrying a convergence risk (see the caution above). With stock Aeron and a hard cutover, whole-cluster swap is simpler and its one hit (~7–8 s) is slightly smaller than node-by-node’s one leader hit (~11.8 s). Node-by-node wins when you need client-transparency or near-zero downtime (followers free + a graceful/RESIGN leader step) — and a custom NameResolver removes the DNS-cache hazard that otherwise argues for cluster-swap. The deciding factor is the leader-election path — not the migration topology.

Production notes

• Do it shard by shard. Each ME shard is its own cluster. Migrate non-critical shards first, watch ~5 minutes, then migrate the rest in staggered batches — never all at once.
• Keep the reduced-tolerance window short. During a node-by-node swap the cluster runs at bare quorum (2/3, no spare). Restart and re-catch-up each stepped node as a follower before the next swap; don’t leave stepped-down nodes offline.
• Gateways stay in both regions so external clients keep connectivity to either region throughout and after the migration.

See also: Aeron® Cluster Placement, Cluster Standby and Multi-AZ HA Design, Client-Cluster Communication (the NewLeaderEvent and client reconnect), and Leader Placement and Preferred-Leader Control.