Tuning Graceful Step-Down Time

A planned leader handoff — a preferred-leader reclaim, a maintenance step-down — is a different performance problem from an unplanned leader death. There is no failure to detect, so the 10 s leader.heartbeat.timeout that dominates unplanned failover never enters the budget. What is left is a ~266 ms client-visible outage on stock settings, and five configuration knobs that cut it to ~57 ms with no code change.

This page is the measured answer to “which knobs can I turn, and what does each one do to the client-observed handoff time?”

For the byte-level election internals — terms, canvass, vote records, the end-of-stream frame flag on the wire — see The Aeron Files. This page is the operational view: what to turn, what it buys, and what it does not.

The metric of record

Everything here is measured as serviceGapMs: the client-observed outage on one client clock, from the last reply served by the old leader to the first committed reply from the new leader.

That definition matters, because it deliberately excludes the react gap — the ~4 ms between the RESIGN toggle being latched and the old leader noticing it on its next slow-tick poll. During the react gap the old leader is still serving the client normally, so it is not outage. Quote the outage, not the end-to-end number, which is ~4 ms longer.

Where the milliseconds go

Four roles on one chrony-aligned clock. Each bar is a measured span; each segment inside a bar is a measured sub-stage, coloured by how the time is spent — burning CPU, waiting out a tunable timer, on the network, or parked in a poll. Turn a knob and the spans it controls move.

The thing worth looking for: almost nothing here is CPU or wire. One ~2 ms sliver of genuine CPU (logBuffers allocation) and a handful of sub-2 ms network trips. Everything else is a timer or a parked poll — which is exactly why configuration buys so much, and why it runs out where it does.

Preset: Zoom 8 px/ms arrows react 4 ms → client outage 81 ms · end-to-end 85 ms

node2 NEW leader node0 OLD leader (demoting) WAIT slice node1 follower client client-perceived outage

sub-stage class (how the time is spent): CPU spin/bound tunable timer wait network round-trip network one-way parked poll ⟳ = a tunable knob sets this span's length · arrows: solid ⇄ = round-trip, dashed → = one-way

Click a block to break down its milliseconds. Hover a cross-node arrow to see the interaction and the knob that gates it.

Tunable knobs — turn one to see the spans change

Reading the combine rule: Spans within one lane are serial on that node's consensus-module thread, so they add. Lanes overlap in wall-clock — all anchored at t=0, the RESIGN toggle. The client outage runs from TX step-down (t≈4 ms, the last reply the old leader served) to the first committed reply from the new leader: that is the serviceGap metric of record. The end-to-end badge adds the 4 ms react gap before TX, during which the client was still being served — so quote the outage, not end-to-end.

⚠️ Every absolute here is "as instrumented." All arms were measured with RESIGN_PROBE=true, i.e. with System.out.println stamps on the consensus-module hot path. A probes-ON vs probes-OFF control measured that instrumentation at ~9 ms (ON 68/66/65, mean 66.3 vs OFF 58/58/56, mean 57.3 — non-overlapping ranges, the whole delta inside learnNewLeaderMs). So the real multi-AZ floor is ~57 ms, not ~65 ms; single-AZ was never measured probes-off, so no probes-off number is claimed for it. The levers are unaffected — every A/B was probes-ON vs probes-ON, so the deltas stand; only the absolute floor moves.

Knobs are NOT additive — read the badge before quoting a combo. When you pick a combo that was never run end-to-end, per-span values come from measured single-knob A/Bs and are summed: an estimate, flagged in the badge. Two measured masking effects show why summing overstates — setup=20 alone gave −5 ms, not the −46 ms its span-shrink predicts (slow detection masks it until heartbeat.timeout comes down too, which is why you always change both), and heartbeat=5 gave −6 ms alone but zero on top of slow.tick=1ms. Estimates with heartbeat still at 100 ms are the least trustworthy.

The recommended package

# ---- on every CLUSTER NODE jvm ----
aeron.publication.heartbeat.timeout=10000000     # 10ms — ns integer, see the parse trap below
aeron.publication.setup.timeout=10000000         # 10ms — ns integer
aeron.cluster.idle.strategy=noop
aeron.conductor.idle.strategy=noop
aeron.sender.idle.strategy=noop
aeron.receiver.idle.strategy=noop
aeron.client.idle.sleep.duration=1ms             # service-container client-conductor; NOT an idle.strategy
aeron.cluster.slow.tick.interval=1ms             # multi-AZ; use 500us for single-AZ

# ---- on the CLUSTER CLIENT jvm (embedded SHARED driver) ----
aeron.shared.idle.strategy=noop

Cost: ~5 spinning cores per node plus 1 client core (standard practice for Aeron Cluster), slightly higher steady-state SETUP/heartbeat traffic on every publication, and a marginally faster slow-tick poll.

configuration	multi-AZ	single-AZ
Stock (all defaults)	266 ms	260 ms
Recommended package, as instrumented	~64 ms	~51 ms (best rep 46 ms)
Recommended package, instrumentation removed	~57 ms ← the real number	not measured probes-off

The five knobs that work

Best first. Every figure measured on AWS (pinned cores, chrony sub-µs, 3–5 reps per arm).

1. `aeron.publication.heartbeat.timeout` — 100 ms → 10 ms

266 → 185 ms at 20 ms (−81 ms), scaling roughly linearly down to 10 ms. It attacks detection (~115 ms → ~22 ms).

A gracefully-closed log publication’s end-of-stream flag only ships on the next driver heartbeat, and followers cannot enter the election until it arrives. This is a media-driver timer — which is why every cluster timer measured flat. It is the single biggest lever, and it is not in the cluster configuration at all.

2. `aeron.publication.setup.timeout` — 100 ms → 10 ms

−142 ms when paired with knob 1 (both at 20 ms → 124 ms). Only −5 ms on its own.

Each election creates a fresh log publication, and followers must form its Image via SETUP ↔ StatusMessage, re-emitted on this timer. The pairing is the point: while detection is still ~115 ms, the followers’ cold-image rounds complete inside the detection shadow, so shrinking them changes nothing the client can see. Synergistic, not additive — always change both.

3. Idle strategies → `noop`

−13 ms multi-AZ (105.4 → 92.0, t = −3.19, 95 % CI [−23.1, −3.7]); −17 ms single-AZ (92.2 → 75.2, Mann-Whitney p = 0.016).

This one is diffuse — it collapses no single phase. It removes the ~1 ms Backoff park at every hop of the serial multi-agent election chain (JFR confirms zero in-window parks on the hot path afterwards).

The bigger win is variance: stdev collapses 6×, 15.9 → 2.5 ms. If you are writing an SLA rather than chasing a median, this is the knob that matters most.

4. `aeron.client.idle.sleep.duration` — 16 ms → 1 ms

−16 ms (97 → 81 ms), landing entirely in leaderInit → awaitServicesReady → awaitImage (8.3 ms → 0).

ClusteredServiceContainer builds its Aeron client with a default context whose conductor idles on SleepingMillisIdleStrategy(16 ms). Easy to miss twice over: it is a sleep duration, not an idle.strategy, and aeron.cluster.idle.strategy does not cover it.

5. `aeron.cluster.slow.tick.interval` — 10 ms → 1 ms

−13 ms (82.0 → 68.7, ranges do not overlap), landing in learnNewLeaderMs (73.0 → 58.0).

It gates slowTickWork → checkSessions → sendNewLeaderEvent — the NewLeaderEvent notify cadence. The subtlety: the T0→T1 prep dwell itself does not shrink. The win is in delivery timing, not prep duration — the client simply learns who the new leader is ~15 ms sooner.

Where each knob bottoms out

knob	knee	evidence
`heartbeat.timeout`	10 ms	100 → 20 → 10 → 5 ms gives detection 115 → 34 → 22 → 16 ms (still linear at 5 ms), but hb=5 measured zero gain once slow.tick=1 ms is applied (64/65 vs 64.7 — masking). 10 ms is the practical value.
`setup.timeout`	10 ms	below that it hits the ≥1-RTT-per-round floor and doubles SETUP traffic for a sub-millisecond return
`slow.tick.interval`	1 ms multi-AZ / 500 µs single-AZ	4-value sweep × 2 environments: multi-AZ 10 ms → 77.0, 1 ms → 64.7, 500 µs → 60.7 (noise, overlaps), 100 µs → 64.3 ⇒ floors at 1 ms. Single-AZ 62.3 / 57.3 / 51.3 / 51.3 ⇒ keeps gaining through 500 µs.

Co-location is worth ~7–15 ms — and it is rung-dependent

Not a knob, but measured and real: putting the 3 voters and the client in one AZ is worth ~7–15 ms, and how much depends on where else you are. multi-AZ minus single-AZ is 14.7 ms at slow.tick=10 ms but only 7.4 ms at 1 ms.

It is invisible at stock (260 vs 266 ms) because the ~250 ms fixed cost swamps it. Distance only surfaces once the timer and park costs are stripped out — a good illustration of why you cannot rank these knobs independently of each other.

Knobs are not additive — the trap that invalidates most estimates

This is the most important operational caveat on the page, and the reason the timeline above refuses to print a number for combinations that were never actually run.

Two measured masking effects:

setup=20 alone gave −5 ms, not the −46 ms its span-shrink predicts. Slow detection masks it.
heartbeat=5 gave −6 ms alone but exactly zero on top of slow.tick=1 ms. Once slow-tick has compressed the notify path, the detection saving no longer reaches the client gap.

Measure the combination you intend to ship. A spreadsheet that sums per-knob deltas will overstate the result, sometimes by 60 ms.

The 15 knobs that do nothing — do not re-test these

All A/B tested on AWS with flag carriage verified live on the JVM command line. Ranges overlap, so: inert.

knob	tried	result	why it fails
`aeron.pending.setups.timeout`	1000 → 50 ms	82.0 → 76.3, overlaps	a retry-slot deadline; the frame trace shows the first SETUP ↔ SM always succeeds, so it never fires
`aeron.cluster.election.timeout`	1 s → 500 ms	84.0 ≈ 82.0	a failure-detection ceiling; the graceful fast path finishes in ~110–130 ms and never approaches it. 200 ms breaks the handoff — do not go there.
`aeron.cluster.election.status.interval`	100 → 1 ms	flat	the ballot uses the unanimous fast path
`aeron.cluster.leader.heartbeat.interval`	200 → 20 ms	flat	acks arrive by physical catch-up, not by waiting out a tick
`aeron.cluster.leader.heartbeat.timeout`	10 s	off the hot path	a fallback failure deadline — this is the knob that dominates unplanned failover, and it is irrelevant here
`aeron.rcv.status.message.timeout` (node and client)	200 → 20 ms	flat	paces the steady-state keepalive SM, not the setup-eliciting one
`aeron.archive.idle.strategy` (+ recorder/replayer)	Backoff → busyspin	flat	see the structural rules below
`aeron.archive.control.idle.strategy`	Backoff → noop	81 → 79 (noise)	third-order; shaves ~385 µs off `startLogRecording` only
`aeron.client.awaiting.idle.sleep.duration`	1 ms → 0 (busy-spin)	65.3 → 63.0	governs only ~1 ms of an ~8 ms span
`aeron.driver.async.executor.threads`	1 → 0 (inline)	64.7 → 64.3	structural rule 1
`aeron.timer.interval`	1 s → 10 ms	64.7 → 63.3	at low load `decRef()` takes the already-drained fast path, so the 1 s timer never gates end-of-stream. ⚠️ Latent under-load risk — see the traps below.
client-side `aeron.publication.{heartbeat,setup}.timeout`	100 → 20 ms	+2 ms	the client’s re-establish is not the cold connect these timers govern
`aeron.term.buffer.sparse.file` / log-channel `term-length`	true ↔ false / 64m → 1m	flat	buffer allocation is ~1 ms; never the cost
`aeron.cluster.clock` = Nanosecond	—	flat	—
`aeron.spies.simulate.connection`	false → true	flat	the spy already links immediately; nothing to promote

SUSPEND-then-RESIGN deserves its own line: measured a wash to a slight regression. It is a correctness lever (it gives you a deterministic caught-up winner), not a latency one — it relocates the replay drain into client downtime rather than removing it.

Traps that will corrupt your measurements

aeron.publication.{heartbeat,setup}.timeout and aeron.pending.setups.timeout need a bare nanosecond integer. They are read via Long.getLong → Long.decode, which cannot parse a "10ms" suffix and silently falls back to the 100 ms default with no error. A full JFR run was wasted at stock timers this way. Cluster and idle properties (slow.tick.interval, election.status.interval, client.idle.sleep.duration) use getDurationInNanos and accept suffixes fine.
Verify a flag’s effect, not its carriage. A property can sit on the JVM command line and still not take effect (see trap 1). Confirm via a probe or a dwell measurement that the intended behaviour actually changed.
System.out.println probes are not free — ~9 ms here. Quote absolute durations from a wait-free tracer, JFR, or client-side nanoTime; use printlns only for ordering and cross-node epoch alignment.
aeron.timer.interval (default 1 s) is a latent under-load risk. decRef() sets isEndOfStream only if the publication is already drained; otherwise end-of-stream is set inside onTimeEvent, gated by this 1 s timer. A 1 msg/ms rig always takes the drained fast path — which is why it measured inert. Under load, or with a backlog at resign, detection could be gated by 1 s. Not measured under load; 10 ms is cheap insurance for a loaded deployment.
Knobs are not additive (above). Measure the combination you intend to ship.

What remains after all configuration

The residual ~57 ms is: at least one wire round-trip (vote, appendPosition ack, first commit) + at least one poll tick per hop on already-spinning threads + serial cross-agent hop counts + ~2 ms of genuine CPU (logBuffers allocation, the only on-CPU sliver in the whole failover) + spans that are overlapped away (terminal quorum is max(follower catch-up, node2 replay+init) − node2 replay+init).

Only two changes go below it. Both are code, and both attack hop count rather than timers:

Warm or reuse the per-term log publication. Every election currently creates a fresh log publication (new sessionId, therefore cold). Removing that removes the follower cold-image SETUP rounds, node2’s ~2 ms logBuffers allocation, and most of joinLogAsLeader (~4.3 ms tracer-measured).
Warm ingress pre-connect on the client. Hold warm ingress publications to all members before the handoff, so onNewLeader needs no fresh publish (~1.4 ms of measured Aeron work today, plus the ~6.7 ms driver duty-cycle wait behind it).

Putting it in proportion

event	cost	dominated by
Planned step-down, tuned	~57 ms	irreducible round-trips + hop counts
Planned step-down, stock	~266 ms	`publication.heartbeat.timeout` (a driver timer)
Unplanned leader death, balanced tier	~1.5 s	`leader.heartbeat.timeout` (a cluster timer)
Unplanned leader death, stock	~11 s	`leader.heartbeat.timeout` = 10 s

A planned handoff is ~40× cheaper than an unplanned death even before you tune it, and ~190× cheaper after. That gap is the entire argument for driving preferred-leader reclaims through a graceful path rather than by killing the leader — see Leader Placement and Preferred-Leader Control.

Two consequences worth internalising:

The knobs do not overlap between the two regimes. Tuning leader.heartbeat.timeout does nothing for a planned handoff; tuning publication.heartbeat.timeout does nothing for an unplanned death. They are separate budgets with separate dominant terms.
Reclaim frequency changes the calculus. At ~57 ms a preferred-leader reclaim is cheap enough to run routinely. At 266 ms it is still cheap, but it is a visible latency event on a busy hot path — worth scheduling rather than firing on every AZ health blip. Hysteresis still applies either way.

See also: Leader Placement and Preferred-Leader Control (who wins the election and how to trigger the handoff), Tuning Cluster Failover Time (the unplanned-death budget), and Client-Cluster Communication (the new-leader event and client reconnect).