rsx-cast

The retransmit source is the WAL.

The 16-byte header + repr(C) payload the matching engine writes to its write-ahead log is the same bytes on the UDP wire and the same bytes on the TCP replay stream. No serialize step, no encode step, no length-prefix wrapper. Every other option puts a framing layer on top of records that are already framed; casting skips it.

📡

UDP frame (live)

💾

WAL record (disk)

🔁

TCP replay (cold)

↑ bitwise identical — one bytestream, three uses ↑

Round-trip latency vs the field (loopback p50, 128 B = size_of FillRecord)

casting (rsx-cast)

~10 µs

raw UDP (floor)

~10 µs

MoldUDP64 (Nasdaq)

~10 µs

TCP_NODELAY

~14 µs

SoupBinTCP

~14 µs

KCP (turbo+spin)

~21 µs

Quinn / QUIC

~37 µs

Aeron (UDP)

~305 µs

Bars use a compressed (≈√) scale so the 10 µs cluster stays readable next to Aeron's 305 µs networked path. casting ties the raw-UDP floor and MoldUDP64, beats both TCP protocols, both userspace-RUDP options (KCP, QUIC), and Aeron's UDP path. The only faster numbers in the survey are shared-memory IPC paths (Aeron IPC ~830 ns, Chronicle sub-µs) — not network transports, and they carry no WAL.

Bench, not live: Criterion loopback microbenches, client+echoer pinned to cores 2/3, Rust release. Reproduce: cargo bench -p rsx-cast --bench compare_all. p50 only; p99 not yet measured. The exchange's real cross-process p50 is ~1 128 µs — dominated by runtime sleep-polls and PG churn, not transport (a ~10 µs slice of that).

Where the ~10 µs RTT actually goes

sendto syscall ~99%

CastSender::send body is ~4.0 µs, of which 99% is the kernel sendto. The protocol's own work — assign seq, CRC32C, copy into the preallocated ring — rounds to zero. That is why casting sits at the raw-UDP floor: there is almost nothing on top of the syscall.

WAL append (in-memory) ~31 ns

Nak / Heartbeat encode ~43 ns

Fill encode ~23 ns

Fill decode ~9 ns

casting one-way (send→recv) ~5.3 µs

Bench (release). Source: README.md "Network stack", rsx-cast/README.md "How fast". Zero heap allocation on the send path — every send is a copy into an already-owned ring slot + one syscall.

Wire format = disk format (16-byte header, then repr(C) payload)

version u8 · @0

_pad0 u8 · @1

record_type u16 · @2

len u16 · @4

_pad1 u16 · @6

crc32c u32 · @8

reserved [u8;4] · @12

payload — repr(C, align(64)), ≤ 65535 B (seq = first u64 of every data record)

CRC32C (Castagnoli, SSE4.2 hardware path, ~1 cycle/8 B) over the payload only. version at offset 0; adding a new record type does not bump it. Little-endian, compile-time enforced. Because there is no separate on-disk schema, the byte you fsync is the byte you retransmit.

NAK retransmit, two tiers (embedded, not a sidecar)

receiver detects a gap → sends Nak(seq)

Tier 1 — hot ring (RAM)

4 K most-recent frames, preallocated. µs to re-send.

↓ on miss

Tier 2 — cold WAL (disk)

pick the segment whose filename seq-range covers the target, scan it (read_record_at_seq). ~10.4 ms @ 10 K records.

Retransmit horizon = WAL retention (4 h default), not RAM.

Aeron ships its cold retransmit as a separate Archive process. casting keeps the audit log and the retransmit cache in the same producer — the protocol invention is small, the packaging difference is the point.

A slow consumer never stalls the producer: it recovers via this NAK path, or — if it falls too far behind — escalates to a full TCP replay and resumes UDP. Senders never pause; there is no flow control.

10.4 ms cold-tier number is a bench (wal_random_read_bench, real SSD); the 4 K / 4 h figures are config defaults.

What happens when a packet drops (seq 3 lost)

TCP — head-of-line blocking

1 2 3 4 5

4 and 5 already arrived but the kernel withholds them until 3 is re-ACKed and resent. The whole stream waits — even records that have nothing to do with the loss.

casting — gap parked, NAK fills it

1 2 3? 4 5

4 and 5 buffer in the reorder ring; the receiver Nak(3), the sender re-sends 3 from its hot ring (µs), delivery resumes in order. Producer never blocked. Tuned for ≤ 0.01% loss on a trusted LAN — on a lossy WAN use QUIC/KCP instead.

casting vs the field — features

	casting	Aeron	MoldUDP64	Quinn / QUIC
Retransmit source	hot ring + WAL	term buffers (RAM)	separate request server	in-flight window
Retransmit horizon	WAL retention (4 h)	~192 MB RAM	server policy	in-flight window
Durability	wire = disk format	separate Archive	external	none
Wire format	16 B repr(C)	32 B + term offsets	20 B header	variable QUIC frames
Serialization step	none (repr(C))	session framing	external	QUIC framing
Congestion control	none (trusted LAN)	none	none	yes (TLS+CC)
Encryption	none — gateway/L3 owns it	none	none	TLS 1.3
Language	Rust	Java + C++	any (public spec)	Rust

Source: rsx-cast/compare/README.md. "none" is often a feature here: no congestion control and no encryption are deliberate — casting is a trusted-LAN transport (specs/2/4-cast.md §10.4) and delegates auth to the gateway (JWT/TLS) and the L3 network. For the public internet, the project says use QUIC — that is the right tool for that job, not casting.

When NOT to use casting (it is not fastest-in-general)

✗ Public internet — no TLS, no congestion control → use QUIC
✗ Lossy / high-jitter WAN — NAK storms collapse throughput; tuned for ≤ 0.01% loss → use KCP
✗ Schema that changes often — wire = disk = repr(C) means changes are stop-redeploy events → use protobuf / FlatBuffers
✗ Multi-language consumers — no IDL; hand-write the repr(C) layout per language
✗ One-to-many fan-out today — single sender per stream; v2 multicast is specced, not shipped
✗ Latency-sensitive cold replay — read_record_at_seq is O(N) within a segment (~10.4 ms @ 10 K)

casting is "at the UDP floor for a fixed-record trusted-LAN workload", not a universal winner — and the project says so out loud. Source: rsx-cast/README.md "When NOT to use this".

Every number sourced from repo docs / benches — loopback p50, Criterion release. Reproduce: cargo bench -p rsx-cast --bench compare_all