Host & OS Latency Parameters

The Parameter Reference covers Aeron’s transport knobs. This page is the layer underneath: the host, OS, and JVM settings that decide whether those transport knobs ever get to run on a quiet, predictable core. Almost every multi-millisecond tail that survives correct buffer sizing traces back to one of these — a core dropping into a deep C-state, a clock that sags under load, khugepaged collapsing pages, or an IRQ landing on a remote NUMA node.

The one mental model

Transport tuning keeps the pipe full; host tuning keeps the core deterministic. p99 is variance, and variance is born here — in power-state transitions, scheduler preemption, cache/NUMA penalties, and page-fault stalls. Get this layer flat first, then tune transport on top of it.

How to read the tables

Two things matter for every system knob beyond its latency effect: where it lives and how long it lasts. The persistence model is the part people get wrong.

Persistence class	Set via	Survives reboot?	Notes
GRUB / kernel cmdline	grubby --update-kernel=ALL --args=... then reboot	✅ yes	The only class that requires a reboot to apply. On BLS-based distros (Amazon Linux 2023, recent Fedora/RHEL) grubby is authoritative — editing /etc/default/grub alone is not enough.
sysfs / proc (live)	echo ... > /sys/...	❌ no — resets on boot	Applies instantly to the running kernel. No reboot, no process restart. Lost on reboot unless re-applied.
systemd unit / tuned profile	a unit/profile that writes the sysfs value at boot	✅ yes	The reboot-free way to make a live sysfs change persistent — apply live now, let the unit re-apply on every boot.
process flag	launch script / service args	✅ yes (re-read each start)	e.g. JVM flags. Persistent because the launcher re-execs them — but needs a process restart, not a machine reboot, to take effect.

Reboot vs. restart — they are not the same

Most host knobs (frequency, C-states already in cmdline, THP, IRQ affinity, socket buffers) are live sysfs — no reboot needed. Only adding/removing a kernel cmdline argument forces a reboot. And a few changes (JVM flags, NUMA pin that wraps the process) need the application restarted but not the host. Know which class you’re touching before you schedule downtime.

CPU power: C-states, P-states, frequency

The single biggest source of µs-to-ms jitter on an otherwise idle-looking box is the CPU changing power state. A core that has dropped into a deep idle state takes microseconds to wake; a clock that ramps on demand adds variance to every first message after a quiet gap.

Parameter	Throughput	p50	p99 (tail)	Set via	Persist	Revert
C-states (intel_idle.max_cstate=1, processor.max_cstate=1)	Negligible	Small win — cores don’t sleep between bursts	Large win — eliminates deep-sleep wake latency on the first packet after an idle gap	GRUB cmdline	✅ reboot	grubby --remove-args=... + reboot
CPU governor (performance)	Keeps clocks high under load	Lower baseline	Removes governor ramp lag	cpupower frequency-set -g performance (sysfs)	❌ live	set back to schedutil/ondemand
P-state driver mode (intel_pstate=passive / amd_pstate=guided)	n/a	n/a	Lets the governor — not opaque HWP — own the clock, so a pin actually holds	GRUB cmdline	✅ reboot	remove arg + reboot
Frequency pin (scaling_min_freq = scaling_max_freq)	Slightly below peak burst	Dead-flat baseline	Removes clock variance entirely — every core at one frequency	sysfs per-CPU, or cpupower frequency-set -d F -u F	❌ live	write cpuinfo_min/cpuinfo_max back
Turbo / boost (intel_pstate/no_turbo, cpufreq/boost)	Higher single-core peak	n/a	Turbo raises peak but adds variance; disabling pins to base (lower, but dead-uniform)	sysfs	❌ live	re-enable (no_turbo=0 / boost=1)

The trap: single-core turbo ≠ all-core frequency

A datasheet “max turbo” (e.g. 3.8–4.5 GHz) is a single-core number — reachable only when most cores are idle so the package has thermal/power headroom. The instant you load all cores, the hardware forces everyone down to the lower all-core turbo bin. So “maximum GHz” and “uniform GHz under load” are mutually exclusive: you cannot have every core at the single-core peak simultaneously.

For latency work you almost always want uniform over maximum. A clock that is flat across all cores — even at a lower value — removes a whole class of cross-core variance. The recipe:

1. Release any pin and let the box find its natural all-core bin under full load (measure it — next section).
2. Pin scaling_min_freq = scaling_max_freq at or just below that measured bin, so no core ever has to sag.

Two quantization gotchas

• intel_cpufreq moves in ~100 MHz steps. Ask for 2950 and you get 2900; ask for 3243 and you get 3200. Pin to a step boundary and expect a snap.
• Some SKUs have a hard all-core ceiling below their single-core max. The policy will happily report the higher value while every core still runs the lower one. Always confirm with turbostat under load (below) — never trust the requested scaling_max_freq alone.

Confirming a flat all-core clock (detect + stress)

This is the verification step people skip — and then a “pinned” box quietly runs three different clocks across its cores. The method is: drive every core to 100% busy, then read the actual per-core frequency and check the spread.

Step 1 — read the requested pin

# What the policy is set to (one line per unique min/max pair — expect exactly one):
paste \
  <(cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq) \
  <(cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq) | sort -u
# Driver + governor + turbo state:
cat /sys/devices/system/cpu/cpufreq/policy0/scaling_driver   # intel_cpufreq | acpi-cpufreq | amd-pstate
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor    # expect: performance
cat /sys/devices/system/cpu/intel_pstate/no_turbo 2>/dev/null # 1 = turbo off

The requested pin is necessary but not sufficient — the SKU may not honor it under all-core load.

Step 2 — stress every core and read the achieved frequency

turbostat reports Bzy_MHz — the actual busy frequency each core ran at, which is the number that matters. The cleanest way is turbostat’s own built-in stress subcommand, which loads every logical CPU and samples them in one self-contained command (no stress-ng install needed):

# Preferred — turbostat's built-in all-core load. Pins one busy thread per logical CPU for SEC seconds:
sudo turbostat stress --cpu "$(nproc)" -t 30

# Observe-only (current workload, no added load) with the columns that matter:
sudo turbostat --interval 2 --show Core,CPU,Busy%,Bzy_MHz,Avg_MHz,TSC_MHz,C1%,IPC,IRQ

# Portable fallback if turbostat lacks the stress subcommand — busy-loop per CPU:
for i in $(seq 1 "$(nproc)"); do yes > /dev/null & done
turbostat --interval 1 --num_iterations 10 --quiet
pkill yes        # ALWAYS stop the load afterwards

Read two things in the output:

• The aggregate row (label - in the CPU column) gives the all-core average Bzy_MHz.
• Each per-core row gives that core’s Bzy_MHz. Compute min / avg / max across them.

turbostat column legend

• Bzy_MHz — actual frequency while busy (the key flatness indicator; compare this across cores).
• Avg_MHz — average over the whole sample including idle (lower than Bzy_MHz on a quiet core).
• TSC_MHz — the invariant base/TSC frequency (a fixed reference, not the running clock).
• Busy% — fraction of the interval the core was non-idle (≈100% under the stress load).
• C1% — fraction spent in the C1 idle state (≈0 on a hot busy-spin core; high on idle cores).
• IPC — instructions per cycle; IRQ — interrupts taken in the interval (a busy-spin agent core with IRQs steered away should show a low, stable IRQ count).

The header turbostat prints also echoes the kernel command line, the cpufreq governor, and the current cpufreq driver — a one-glance confirmation that your cmdline (C-state/nohz_full/pstate) and governor pins are actually in force.

Step 3 — apply the spread gate

flat if   (max − min) / avg  ≤  ~1%

Observation	Verdict
All cores within ~1% of each other and of the pin	✅ genuinely flat all-core — pin is honored
A handful of cores sag below the rest under load	⚠️ pin is above the all-core bin → lower the pin to the sagging floor
Every core runs below the requested scaling_max_freq	🔴 hard SKU ceiling — the all-core bin is below your ask; pin at the achieved value, not the requested one
Wide spread only in the first 1–2 samples, then converges	ramp lag on a cold box, not a real difference — warm up ≥5 s before judging

Why the warm-up matters

A box that just went from idle to all-core-busy shows a low first sample purely because the clock is still ramping. Always discard the first couple of turbostat intervals; judge flatness only on the sustained samples.

This same loop is how you discover the natural all-core bin in the first place: run it with the pin released (scaling_max_freq = cpuinfo_max_freq), read the sustained all-core Bzy_MHz, then pin at or just under it.

CPU topology & scheduling

Parameter	Throughput	p50	p99 (tail)	Set via	Persist	Revert
SMT / Hyper-Threading	More logical CPUs for non-critical work	Neutral if hot threads never share a core	Two hot threads on one physical core (SMT siblings) contend → tail spikes; keep critical agents off siblings	/sys/devices/system/cpu/smt/control (sysfs)	❌ live	echo on > .../smt/control
Core isolation & pinning	Higher sustainable max — no run-queue contention	Less context-switch cost	One of the strongest tail levers — see Core Isolation & Pinning	isolcpus/nohz_full (GRUB) + taskset/numactl (process)	mixed	per-method
Tickless + RCU offload (nohz_full=<cpus>, rcu_nocbs=<cpus>)	n/a	n/a	Removes the periodic scheduler timer tick and RCU softirq callbacks from the listed CPUs → no recurring interruption of a busy-spin agent. Pair the CPU list with your pinned cores	GRUB cmdline	✅ reboot	grubby --remove-args=... + reboot
NUMA placement (threads + memory NIC-local)	Removes cross-socket traffic	Cuts memory/PCIe traversal cost	Large — avoids remote-NUMA penalty; the receiver-stall fix lives here. See NUMA & Cache Locality	numactl --cpunodebind --membind wrapping the process	process restart	drop the wrapper
IRQ affinity (NIC IRQs → NIC-local cores)	Keeps RX off busy cores	Lower interrupt-handling jitter	Stops NIC interrupts preempting the receiver agent	/proc/irq/<n>/smp_affinity_list (sysfs)	❌ live	restore previous mask
irqbalance	n/a	n/a	Daemon re-shuffles IRQs at runtime → periodic affinity drift → sporadic spikes; disable for pinned setups	systemctl disable --now irqbalance	✅	systemctl enable --now irqbalance
kernel.numa_balancing	n/a	Removes page-migration scan cost	Stops automatic NUMA page migration from stalling hot threads	sysctl / 99-*.conf	✅ (file)	set back to 1

Confirm SMT siblings aren’t doubled up

Pinning to logical CPU IDs isn’t enough — two of them can be siblings of the same physical core. Map each busy thread to its physical_package_id/core_id and assert no physical core hosts two hot agents. On a quiet, correctly-pinned box every busy Aeron agent owns a distinct physical core.

Order of operations matters when isolating

Migrate kernel threads and IRQs off the target cores (e.g. tuna isolate --cpus=<list>) before the app is pinned onto them — and set the app’s CPUAffinity (systemd unit) then restart so it forks onto the now-clean cores. Reserve a non-critical core (e.g. core 0) for the OS/kernel housekeeping rather than isolating literally everything. Doing the isolation after the app is already running drags the live agent threads along with the migration.

Memory: hugepages & THP

Parameter	Throughput	p50	p99 (tail)	Set via	Persist	Revert
Transparent HugePages (THP) mode	Fewer TLB misses on large heaps	Slightly lower with hugepage-backed heap	always lets khugepaged collapse pages process-wide → periodic stalls; madvise is calmer; never is the most deterministic	/sys/kernel/mm/transparent_hugepage/{enabled,defrag} (sysfs)	❌ live (or GRUB transparent_hugepage=never)	echo madvise > ...
JVM +UseTransparentHugePages	TLB win on the heap	—	With always/madvise the JVM madvises its heap into THP; disabling (-UseTransparentHugePages) takes the heap off THP entirely	JVM flag in launch script	process restart	flip flag back, restart
Explicit hugepage pool (nr_hugepages)	Required for kernel-bypass (DPDK) mbuf pools	—	Pre-reserved, never collapsed → no khugepaged jitter	sysfs per NUMA node	❌ live	set pool back to 0
vm.swappiness=0	—	Avoids reclaim stalls	Stops the kernel swapping hot pages under memory pressure	sysctl	✅ (file)	default 60
vm.compaction_proactiveness=0	—	—	Disables proactive background memory compaction — its page-migration scans steal cycles and fault hot pages mid-run	sysctl	✅ (file)	default 20
vm.max_map_count (raise)	Allows many mapped buffers/term files	—	Prevents mmap failures under many streams	sysctl	✅ (file)	default

THP + ZGC: a deliberate choice

The common guidance “avoid THP for latency-sensitive apps” targets the always mode, where khugepaged scans and collapses pages across the whole process and injects jitter. madvise mode only touches regions an app explicitly requests, which is the calmer flavour a hugepage-aware GC (e.g. ZGC with -XX:+UseTransparentHugePages and -XX:+AlwaysPreTouch) uses on purpose to back its heap. If you are hunting a tail, the clean experiment is to disable it on both layers — system sysfs never and the JVM flag — and A/B the receiver-side segments. Disabling sysfs alone leaves the already-mapped heap on THP until the JVM restarts: live echo never stops future collapsing, but the running heap keeps its huge pages until the process is bounced.

THP ≠ the explicit hugepage pool

HugePages_Total (the nr_hugepages pool, e.g. reserved for DPDK) is a separate reservation from THP. Disabling THP does not touch it, and kernel-bypass still needs it. Don’t conflate the two when reading /proc/meminfo.

Network & socket buffers

These overlap with the transport-layer receive-path buffers; here they’re the OS-level limits that cap what the application can request.

Parameter	Throughput	p50	p99 (tail)	Set via	Persist	Revert
net.core.rmem_max / wmem_max	Headroom to absorb bursts without drops	—	Too small → RX-ring/socket overflow → NAK recovery spikes; size to several × BDP for bursty fan-in	sysctl / 99-*.conf	✅ (file)	distro default (~212 KB)
net.core.netdev_max_backlog	Deeper per-CPU ingress queue	—	Prevents drops when packets arrive faster than the stack drains them	sysctl	✅ (file)	default
net.ipv4.ip_local_reserved_ports	n/a	n/a	Reserves the app’s control/data ports so the kernel never auto-assigns them to an ephemeral socket and collides — an availability/jitter gotcha, not a throughput knob	sysctl	✅ (file)	clear the range
NIC ring buffers (e.g. ENA)	Match rings to bandwidth	Correctly sized keeps queuing low	Oversized rings add queue-bloat latency; undersized drop under bursts	ethtool -G	❌ live	prior ring size

JVM flags that move the tail

Flag	Effect	Persist
-XX:+UseZGC (or a low-pause collector)	Sub-millisecond GC pauses → removes the GC contribution to the tail	process flag
-XX:+AlwaysPreTouch	Faults the whole heap at startup → no first-touch page-fault stall mid-run (pairs with hugepages)	process flag
-XX:+UseTransparentHugePages	Madvises the heap into THP — see the THP note above for when to turn it off	process flag
JIT prewarm	Drive representative traffic through all hot paths before going live — cold code + first-touch cause severe early spikes. See Warming Up the JVM	runtime procedure

Clock synchronization

For any cross-host latency measurement (one-way segment timing, phase-aligned bursts) the hosts must share a sub-microsecond time base, or your numbers are noise.

Parameter	Why it matters	Set via	Persist
PTP / chrony with hardware PHC	Disciplines every host to a common ns-grade reference (e.g. a NIC PTP hardware clock) → honest one-way segment math and viable wall-clock burst alignment	chronyd with refclock PHC	✅ service

Note

Verify, don’t assume: chronyc tracking should show a stratum-1 PHC source selected and last-offset / RMS in the sub-microsecond range on every host before you trust a cross-machine latency breakdown.

Persistence & revert cheat-sheet

Layer	Needs reboot?	Needs app restart?	Lost on reboot unless…
C-states, P-state driver mode, isolcpus, nohz_full, rcu_nocbs	✅ (cmdline)	no	— (GRUB persists)
Governor, frequency pin, turbo, THP sysfs, IRQ affinity, NIC rings, SMT	no	no	re-applied by a systemd unit / tuned profile
sysctl values (rmem_max, swappiness, numa_balancing, compaction_proactiveness, …)	no	no	written to /etc/sysctl.d/*.conf
JVM flags, numactl process wrapper, systemd CPUAffinity	no	✅	— (in the launch script / unit)
Explicit hugepage pool	no	no (but DPDK app needs it before start)	re-applied at boot

Apply order at boot

When a host carries the full stack, the dependencies run in this order each boot, with the app started last so it forks onto an already-quiet machine: kernel sysctl / THP / governor → PTP clock → IRQ & eBPF stamping setup → application. Encode it with systemd Before=/After= (e.g. the THP-disable and governor oneshots are Before= the app unit) so a reboot reproduces the tuned state without manual steps.

Summary

• Flatten the host before tuning transport. C-states limited, governor performance, a measured frequency pin, SMT siblings clear, IRQs and memory NIC-local. These attack variance at its source.
• Know your persistence class. Only kernel-cmdline changes need a reboot; most knobs are live sysfs, and a few need an app restart but not a host reboot.
• Never trust a frequency pin you haven’t stress-confirmed. Load all cores, read Bzy_MHz, gate on a ≤1% spread — the requested scaling_max_freq can silently differ from the achieved all-core clock.
• THP is two layers (system mode + JVM flag) and changing it cleanly means touching both, then restarting the JVM so the live heap actually leaves THP.
• Silence the recurring interrupters. Beyond pinning, the cores running busy-spin agents want nohz_full + rcu_nocbs (no timer tick / RCU softirq), IRQs steered elsewhere, and vm.compaction_proactiveness=0 — each removes a periodic source of jitter that a one-time pin can’t.

This foundation is what makes silicon comparable

With every host knob above flattened identically on two machines, a same-workload comparison isolates the remaining difference to silicon + the transport path. In practice a fully host-tuned newer CPU shows its biggest win exactly where this page targets — the deep-burst tail (p99/p99.9/max), not the median — because the tuning removed the host-side variance and left the core’s own headroom as the differentiator.

For the Aeron transport knobs that sit on top of this foundation, see the Parameter Reference and the tuning overview.