Raw UDP

Baseline: no reliability, no framing, no CRC, no retransmit. The absolute floor for any protocol built on UDP. Anything more capable than this — ordering, gap detection, retransmit, durability — costs more than the numbers below.

Wire format

There is none. UDP has only an 8-byte transport header (RFC 768):

Offset  Size  Field
0       2     Source port
2       2     Destination port
4       2     Length (header + payload)
6       2     Checksum (optional on IPv4, mandatory on IPv6)

After the 8-byte header the kernel hands the application the raw bytes the sender wrote with sendto(). There is no sequence number, no length-prefix beyond the UDP Length field, no message-vs-message framing across multiple datagrams — every sendto is one datagram is one recvfrom on the receiver.

Protocol

std::net::UdpSocket::send_to / recv_from. OS kernel routes the datagram through the loopback network stack. Nothing else.

Loopback path: user → sendto syscall → kernel socket buffer → loopback NIC driver (virtual) → kernel socket buffer → recvfrom syscall → user.

Guarantees

Dimension	Raw UDP	rsx-cast casting
Delivery	Best-effort (may drop)	Reliable (NAK + WAL retransmit)
Ordering	Unordered (may reorder)	Per-stream FIFO (seq monotonic)
Duplicates	May duplicate	Dedup in receiver via seq
Framing	None (one datagram = one recvfrom)	16-byte WalHeader + fixed-size payload
Integrity	UDP checksum (optional, weak)	CRC32C over each record
Durability	None	WAL on disk, 4 h retransmit horizon
Flow control	None (sender can overrun receiver)	None on the wire; producer stalls on WAL flush-lag; bounded reorder buffer
Connection state	None	seq + NAK list + idle-only heartbeat

Everything in the casting column is layered above the kernel UDP socket. Each row is a cost measured against the raw-UDP floor.

Relation to rsx-cast

casting builds on raw UDP. The cost of casting above this baseline is:

• 16-byte WalHeader framing + CRC32C verification
• send_ring slot write (WAL record caching for hot-tier retransmit)
• NAK handling on a detected gap + an idle-only heartbeat (100 ms)
• Sequence number assignment

Measured overhead (loopback, 128 B payload):

raw UDP RTT      8.90 – 11.01 µs  (compare_all::raw_udp_128b, re-run 2026-07-01)
casting RTT      8.36 – 10.47 µs  (cast_rtt_bench cmp_rtt_fill_echo, re-run 2026-07-01)
casting send body    ~4.10 µs     (one-way; cast_send_breakdown_bench, 2026-05-24)
  └─ sendto syscall: 4.07 µs (99.4%)
  └─ userspace (CRC32C + framing + ring copy): ~26 ns

The earlier "~2 µs" raw-UDP baseline claim was wrong for this host — see facts/cast-vs-udp-overhead.md for the full measurement, attribution, and walk-back. Summary: the sendto syscall dominates 99 % of casting's per-send cost; casting's userspace work (CRC32C + WalHeader + ring cache) adds ~26 ns, not microseconds.

Sender + echoer are pinned to cores 2/3 in every RTT bench (core_affinity), which tightened the casting distribution by 10–40% vs the pre-pinning baseline (see the facts file).

Benchmark

benches/compare_all.rs::raw_udp_128b (run with cargo bench -p rsx-cast --bench compare_all). The standalone compare_udp.rs was folded into compare_all.rs in commit 836cfb1.

Two non-blocking sockets on 127.0.0.1, both threads busy-spinning. No per-iteration setsockopt. No blocking recv wake-up. 128-byte payload (matches FillRecord). Measures true kernel UDP round-trip.

Published numbers

Environment	RTT P50
Linux loopback, this host, non-blocking + spin (measured)	~9.9 µs
Linux loopback, blocking recv	~5–10 µs
Same-rack 10 GbE, non-blocking	~5–15 µs
Cross-DC WAN	500 µs – 50 ms

The first row is our measured compare_all::raw_udp_128b (2026-07-01), dominated by two sendto + two recvfrom at ~4 µs each. Some published loopback micro-benchmarks quote ~2 µs; that figure did not reproduce on this host, and facts/cast-vs-udp-overhead.md documents the walk-back.

Sources: RFC 768 (UDP), udp(7) Linux man page, facts/syscall-latency.md (local measurement dfe2ef4), facts/cast-vs-udp-overhead.md (the ~2 µs walk-back).

Why not raw UDP for exchange IPC

• No ordering guarantee across reorder buffers.
• No gap detection — a dropped fill is silently lost.
• No retransmit — consumer must implement all reliability.

Every exchange transport that uses UDP (LBM, Aeron, casting) adds reliability on top. The question is how: NAK-based (Aeron, casting, LBM), ACK-based (KCP), or FEC (Solana Turbine).

MoldUDP64

Nasdaq's UDP multicast dissemination protocol. Carries ITCH 5.0 market data feeds (TotalView, BX, PSX). Public specification, freely implementable. The closest published peer to casting's wire shape: a sequence-numbered, NAK-recovered, fixed-header UDP frame with a fan-out delivery model.

Spec: https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/moldudp64.pdf

Why we include it: same protocol family as casting (UDP + seq + NAK recovery), but with multicast fan-out and a separate retransmit channel. Lets us bench framing overhead against a real exchange wire protocol with a known footprint.

Protocol

Wire format — 20-byte downstream header

All multi-byte integers are big-endian (network byte order), unlike KCP's little-endian framing.

Offset  Size  Field            Meaning
0       10    session          ASCII session ID (left-padded space)
10      8     seq              Sequence number of the FIRST message
                                in this packet (1-based)
18      2     msg_count        Number of messages in this packet
                                (0x0000 = heartbeat, 0xFFFF = end-of-session)
20+     var   messages...      Concatenated length-prefixed messages

Each downstream message inside the packet:

0       2     msg_len   Big-endian length of msg_data
2       N     msg_data  Opaque payload (ITCH 5.0 record, etc.)

Packets are sent over UDP multicast to a well-known group/port. A single packet typically carries one message; bursty market events pack multiple. MTU governs the upper bound (Nasdaq uses 1 472 B payload to stay below 1 500 B Ethernet MTU).

Compare casting/WAL: 16-byte header (version:u8 at offset 0, _pad0:u8, record_type:u16, len:u16, _pad1:u16, crc32:u32 — a CRC32C value in the crc32-named field — _reserved:[u8;4]) + one fixed-size #[repr(C, align(64))] payload per packet. No per-packet message-count; one record per UDP datagram by construction.

Reliability: NAK to a separate request server

MoldUDP64 separates dissemination from retransmit:

1. Downstream UDP multicast carries the live stream (one-to-many).
2. Request channel is a separate UDP unicast (sometimes TCP) endpoint that the receiver queries with a MoldUDP64 Request Packet:

0 10 session 10 8 seq First missing sequence 18 2 msg_count How many sequenced messages requested

3. The request server replies on the same downstream multicast group (so other receivers see the retransmission too — same "NAK suppression" property as Aeron multicast).

End-of-session: a packet with msg_count = 0xFFFF tells receivers the stream is done. Heartbeats (msg_count = 0) keep liveness without payload.

No congestion control, no flow control

MoldUDP64 assumes a fixed-capacity multicast fabric. There is no ACK, no window, no sender-side rate limiting. Receivers that fall behind use the request channel to catch up; the dissemination side never slows down.

This matches casting's design assumption (trusted, fixed-capacity LAN). casting likewise has no wire-level flow control — it is unicast and relies on WAL-writer backpressure (producer stalls on flush-lag

10 ms) plus a bounded receiver reorder buffer rather than any receiver-advertised window on the wire.

Latency characteristics

Public Nasdaq feed numbers (ITCH 5.0 / TotalView): - Wire frame overhead: ~20 B + 2 B per message. - One-way LAN latency reported by Nasdaq colo customers: 10–30 µs (NIC-to-NIC, kernel bypass). - The protocol itself adds essentially zero processing — parse header, dispatch payload.

Relation to rsx-cast

Dimension	MoldUDP64	rsx-cast casting
Transport	UDP multicast (1:N)	UDP unicast (1:1)
Byte order	Big-endian	Little-endian (native x86_64)
Header size	20 B (per packet) + 2 B (per msg)	16 B (per record)
Multiple msgs per packet	Yes (msg_count)	No (one record per datagram)
Loss detection	Receiver (seq gap)	Receiver (seq gap)
Retransmit source	Separate request server	Embedded: hot ring + cold WAL
Retransmit channel	Out-of-band UDP/TCP to request server	Same socket (NAK + sendto)
Multicast NAK suppression	Yes (retransmit on group)	N/A (unicast)
Durable archive	External (TotalView Glimpse)	Embedded WAL (4 h)
End-of-session marker	msg_count = 0xFFFF	None (live tail forever)
Designed use	Market data dissemination (downstream only)	Bidirectional order flow + market data

MoldUDP64 is the dissemination half of an exchange feed (downstream only — no order entry). casting handles both directions in one protocol; it bundles the request-server role into the sender via the embedded WAL.

Stronger than casting

• Multicast fan-out is native. One sender, N receivers, zero per-receiver state on the sender side. casting requires one CastSender instance per peer (point-to-point).
• Multiple messages per UDP datagram. Saves header overhead on bursty market events. casting pays a full 16-byte header per record.
• NAK suppression in multicast means a single retransmit recovers loss for the entire receiver group. casting retransmits per receiver.

Weaker than casting

• Retransmit horizon is implementation-defined. Nasdaq's Glimpse service replays the start-of-day snapshot via a separate TCP protocol. casting's WAL is always there, always 4 h deep.
• Big-endian framing costs bswap64/bswap16 on x86_64 every parse. casting is native little-endian.
• Downstream only. No model for order entry — Nasdaq uses OUCH (SoupBinTCP) for that, two protocols where casting has one.

Benchmark

benches/compare_moldudp64.rs::moldudp64_rtt_loopback_128b (run with cargo bench -p rsx-cast --bench compare_moldudp64) — Criterion, loopback, 128 B payload (matches FillRecord), unicast UDP (not multicast). MoldUDP64 stays a standalone bench (its framing server does not fit the compare_all EchoClient trait) but uses the same payload size and core pinning, so the number is directly comparable.

We bench unicast for fair RTT comparison with the compare_all raw_udp / kcp / tcp set. Loopback multicast on Linux is finicky (IGMP, IP_ADD_MEMBERSHIP, route hints) and would measure kernel multicast plumbing rather than the protocol's framing cost — which is what we want to isolate.

Frame: 20 B downstream header + 2 B message-length + 128 B payload = 150 B on the wire per direction. Sequence number incremented on every send; msg_count = 1. The echoer parses the header, validates the seq and message count, extracts the payload, then frames its own MoldUDP64 packet back with the echoer's own seq counter (a fair, full-stack parse + emit on both sides — not a raw byte echo).

Measured p50 on Linux loopback: ~10 µs (2026-05-24) — at the raw-UDP floor, since the header parse on both sides is tens of ns and the ~4 µs sendto/recvfrom syscalls dominate. Not re-run this session.

Sources

• https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/moldudp64.pdf (official spec)
• https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHspecification.pdf (ITCH 5.0, the payload format)
• https://github.com/martinsumner/moldudp64 (Erlang reference implementation, MIT)
• https://www.fixtrading.org/standards/ (FIX is not MoldUDP, but the request/dissemination split is the same pattern)

Aeron

Open-source Java/C++ reliable UDP transport by Real Logic (Martin Thompson, Todd Montgomery). The direct design ancestor of rsx-cast casting. Widely deployed in HFT and trading systems; acquired by Adaptive Financial Consulting in 2022.

• Repo: https://github.com/aeron-io/aeron (Apache-2.0)
• Wire spec: https://github.com/real-logic/aeron/wiki/Transport-Protocol-Specification
• Rust bindings (used by our bench): https://github.com/gsrxyz/rusteron

Wire format

Aeron's frame layout is term-buffer-oriented. Each stream has three rotating 64 MB term buffers (configurable via term-length). A position in the stream is term_id × term_length + term_offset. Loss detection, flow control, and replay all key off of position rather than a flat sequence number — the term-rotation abstraction lets the receiver reclaim memory aggressively while keeping a window of replay-able data.

Data frame header (32 bytes):

 0-3    frame_length      (header + payload, little-endian)
 4      version
 5      flags             (FRAGMENT_BEGIN, END, EOS)
 6-7    type              (DATA=0x01, PAD=0x02, NAK=0x03, SM=0x04, …)
 8-11   term_offset       (byte offset within the term buffer)
12-15   session_id
16-19   stream_id
20-23   term_id
24-31   reserved_value (8 bytes)
32+     payload

Encoding: little-endian throughout.

casting difference. Our WalHeader is 16 bytes (the on-disk WAL record header doubles as the wire header) with a flat u64 seq, a u16 record_type, a u16 len, and a CRC32C. No term_id / term_offset / session_id. Trade-off:

• Aeron's term layout makes replay zero-copy from RAM in large strides — but the retransmit horizon is whatever fits in the term buffers (default ~192 MB / stream).
• casting's flat seq + WAL file layout makes the disk file the retransmit horizon (4 h retention by default). Slower per retransmit (random-access disk read), but the horizon is measured in hours of traffic rather than megabytes.

Loss detection: NAK from receiver

Aeron is NAK-based. The receiver tracks the highest contiguous position and detects a gap when an out-of-order frame arrives at a higher (term_id, term_offset). It sends a NAK back to the sender naming the missing range.

• Unicast: NAK sent immediately. Single receiver, no implosion risk.
• Multicast: NAK sent after a randomized backoff so that only one receiver per gap actually sends the NAK ("NAK suppression"). Prevents NAK implosion.
• The sender retransmits from its in-memory term buffer.

The model is identical in casting: receiver detects a gap on seq, sends Nak{from_seq, count} back to the sender, sender retransmits. The retransmit path is ~1 RTT. casting is unicast only — no NAK suppression backoff because there can be only one receiver per casting stream.

Retransmit horizon

This is where the two protocols diverge most.

Aeron: retransmit comes from the term buffer the sender still has in memory. Once a position has been overwritten by the rotating term buffers, the live retransmit path can't recover it. For durable replay, Aeron Archive (a separate component) records streams to disk; an archive replay subscription pulls from the archive instead of the live publication. This is a clean separation — the live wire protocol is RAM-bounded and fast; the archive is a different service with its own SUBSCRIBE / REPLAY protocol.

casting/DXS: retransmit is two-tier in the same component.

1. Hot tier: a 4096-slot pre-allocated send ring inside CastSender. NAKs for recent sequences are served from this ring with zero allocation, zero disk I/O.
2. Cold tier: a NAK whose from_seq predates the hot ring falls through to read_record_at_seq on the WAL file. Retransmit horizon = WAL retention (default 4 h).

The WAL is the source of truth for the application and the retransmit reservoir for the transport. There is no archiver sidecar — the producer process owns its own durability.

Flow control

Aeron has multiple flow-control strategies (max, min, tagged) configurable per publication. The sender's position is capped at min(receiver positions) + window where window defaults to half the term length. Receivers send StatusMessage frames advertising their consumption position; the sender uses these to compute the send-side window.

casting has no wire-level flow control — the receiver sends no window-advertisement frame, and the only control record on the wire is a NAK (on a detected gap) plus an idle-only heartbeat. An earlier StatusMessage/receiver-window mechanism was removed in commit 87b223e; casting is single-receiver-per-stream and handles overrun one layer up instead: the WAL writer stalls the producer when flush lag exceeds 10 ms or its buffer fills, and the receiver's bounded reorder buffer caps how far ahead the sender can run. This is a real narrowing vs Aeron — casting cannot throttle a fast sender to a slow receiver mid-stream on the wire; it relies on the WAL-writer backpressure and the trusted-LAN capacity assumption (spec §10.4).

Durability: integrated vs sidecar

	Aeron	casting/DXS
Durability	Aeron Archive (separate process / API)	WAL embedded in CastSender
Sender startup	Connect to driver, no disk	Open WAL file (mmap'd)
Crash recovery	Replay from archive	Replay WAL from last tip
Wire = disk?	No (term buffer vs archive recording format)	Yes (WalHeader + payload, identical bytes)

casting collapses the archive into the protocol. The WAL file you write to disk is the wire format — dd if=/path/to/wal | nc would be a syntactically valid replay stream. This is the "wire = disk = stream" claim that motivates the design.

Media driver

Aeron is structured around a separate media driver process that owns all UDP sockets and shared-memory term buffers. Applications communicate with the driver via lock-free SPSC rings in shared memory (/dev/shm/aeron-$USER/cnc.dat). This gives:

• Multiple clients sharing one driver → one set of sockets, one set of term buffers, lower per-client overhead.
• A driver crash doesn't take out client state immediately (client conductors detect via keep-alive timeout).
• An extra IPC hop on every send and every receive (app → driver shm → kernel UDP → driver shm → app).

casting has no driver. CastSender::send() calls sendto() directly from the application thread. Cost: no IPC hop, but each process owns its own UDP socket and WAL file. Suits RSX's tile architecture (one process per role, pinned thread, dedicated socket).

Performance

Published Aeron numbers (Real Logic / AWS, 2025)

Source: https://aws.amazon.com/blogs/industries/aeron-on-aws-2025-performance-benchmark-results/

Hardware: AWS c6in.16xlarge (Ice Lake, ENA networking, kernel bypass disabled in the "Open Source" column).

Load	Percentile	Open Source	Premium (kernel bypass)
100 k msg/s	P50	21–22 µs	24–25 µs
100 k msg/s	P99	32–43 µs	29–30 µs
1 M msg/s	P50	30–35 µs	30–31 µs
1 M msg/s	P99	57–84 µs	39–40 µs

These are bare-metal-class numbers: dedicated cores per driver agent, large c6in instance, isolated load generators, RTT computed from the embedded message timestamp (in-handler).

Our local bench

rsx-cast/benches/compare_aeron.rs — loopback ping/pong over Aeron UDP, 64-byte payload, embedded media driver in the same process, no core pinning.

Setup	P50 RTT	P99 RTT (approx)	Note
Aeron UDP loopback (this bench, 6-core box, no pinning)	~305 µs	~570 µs	criterion total closure time
Aeron IPC (shared memory, this bench)	~830 ns	~1 µs	non-default; see source for caveat
casting RTT (cast_rtt_bench, same box)	~9.3 µs	n/a	two CastSender/Receiver pairs, loopback (re-run 2026-07-01)
casting send body (cast_send_breakdown_bench)	3.87 µs	n/a	sendto-side only
Aeron AWS open source (c6in.16xlarge)	21–22 µs	32–43 µs	published, pinned cores

Why our Aeron UDP number is 10–30× worse than the published one: on a 6-core machine without core pinning, the driver agent thread + PONG echo thread + PING ping thread + criterion measurement thread + OS background tasks oversubscribe the scheduler. The driver's idle strategy spins, but every preemption costs us hundreds of microseconds of round-trip. The IPC variant doesn't suffer because the kernel UDP path drops out of the critical section. This is not Aeron's protocol overhead — it is our environment.

Why casting RTT is lower in our setup even though Aeron is generally faster in production: casting has no driver IPC hop. On loopback with all threads in one process, sendto() direct from the application is faster than going app → driver SHM → sendto → driver SHM → app by exactly the SHM-ring + scheduler-wakeup cost.

In a properly resourced deployment (≥8 cores, pinned, real NIC, sustained load), Aeron's published numbers reflect what the protocol actually does. Treat our 305 µs as a "laptop-class" data point, not a competitive benchmark.

Guarantees comparison

Property	Aeron	casting/DXS
Reliable delivery	Yes	Yes
Loss detection	NAK (receiver)	NAK (receiver)
Retransmit source	term buffers (RAM, ~192 MB default)	hot ring (4096 slots, RAM) + cold WAL (disk)
Retransmit horizon	term-buffer lifetime (seconds)	WAL retention (4 h default)
Durability	Aeron Archive (separate process)	WAL embedded in sender
Wire = disk	No	Yes
FIFO per stream	Yes	Yes
Multi-receiver	Yes (UDP multicast, multi-destination cast)	No (unicast only)
Flow control	Configurable (max / min / tagged)	None on the wire; WAL-writer backpressure + bounded reorder buffer
Congestion control	Optional (CUBIC)	None
Frame header	32 bytes	16 bytes (WalHeader)
IPC topology	Driver process + clients via SHM rings	None (sendto direct from app thread)
Session setup	SETUP / handshake	None (sendto, zero setup)
Trust model	Configurable (TLS in v1.45+, raw UDP otherwise)	Trusted LAN only (no auth, no encryption)
Language (impl)	C++ media driver, Java client, C client	Rust (native)
Production deployments	Decades in HFT (LMAX, citadel, exchanges)	RSX exchange (this repo)

Where Aeron is genuinely more capable

casting simplified Aeron for one trust assumption (LAN), one topology (unicast), and one language (Rust). Honest side-by-side:

• Multicast / multi-destination cast. Aeron does fan-out at the wire level; casting fans out at the sender (one socket per receiver). For 100 receivers, Aeron multicast sends one packet; casting unicast sends 100.
• Maturity. Aeron is decades old, with production deployments at the largest exchanges. casting is new.
• Wire-level flow control. Aeron lets you pick min/max/tagged per publication; casting has none on the wire — a fast sender can outrun a slow receiver until the reorder buffer or WAL-writer backpressure intervenes.
• TLS option. Aeron 1.45+ supports DTLS; casting delegates TLS to the layers around it (L3 firewall, gateway).
• Position abstraction. Aeron's term-based position model is more flexible for replay/seek operations against large in-memory windows; casting's flat seq + WAL is simpler but doesn't support sub-record seeking.

Where casting is intentionally narrower

• No archiver. The producer process is its own archive. One fewer service to deploy, monitor, recover.
• No driver. No IPC hop, no shared-memory ring between app and transport. Suits a single-process-per-role tile architecture.
• Rust-native. No JVM, no GC pauses, no JNI / SBE layer.
• One trust model. Trusted LAN. The system-spec (specs/2/4-cast.md §10.4) delegates auth to the gateway (JWT) and the L3 network (firewall, VPC). casting is intentionally unauthenticated.

Running the bench

cargo bench -p rsx-cast --bench compare_aeron

Prerequisites (Debian/Ubuntu):

sudo apt install -y cmake libclang-dev clang uuid-dev libbsd-dev

The rusteron-client / rusteron-media-driver crates are configured with features = ["precompile", "static"] in Cargo.toml. This downloads a precompiled Aeron C driver binary from the rusteron release artifacts on first build (no cmake-of-Aeron required at compile time, though Debian 12's stock cmake 3.25 wouldn't satisfy Aeron's >=3.30 requirement anyway). System libs libuuid and libbsd are needed at link time.

The bench source documents an IPC variant (bench_aeron_ipc) that is intentionally not in the default criterion group — running both UDP and IPC variants in one process triggers a C-side MediaDriver has been shutdown race during driver teardown/relaunch. Smoke-measured separately, Aeron IPC RTT is ~830 ns on this hardware.

Sources

• Aeron repository: https://github.com/aeron-io/aeron
• Transport spec: https://github.com/real-logic/aeron/wiki/Transport-Protocol-Specification
• "Aeron: Open-source high performance messaging" — Martin Thompson, Strange Loop 2014: https://www.youtube.com/watch?v=tM4YskS94b0
• LMAX Disruptor paper (the design lineage that produced Aeron): https://lmax-exchange.github.io/disruptor/disruptor.html
• AWS 2025 benchmark: https://aws.amazon.com/blogs/industries/aeron-on-aws-2025-performance-benchmark-results/
• Real Logic blog (general Aeron coverage): https://www.real-logic.co.uk/
• rusteron (Rust bindings used by our bench): https://github.com/gsrxyz/rusteron
• Adaptive Financial Consulting acquisition (2022): https://weareadaptive.com/2022/09/06/adaptive-acquires-real-logic/

KCP

Open-source reliable ARQ protocol over UDP by skywind3000 (Rui Kong). ~1 000 LOC of C in a single header pair (ikcp.h / ikcp.c). MIT-licensed. Widely deployed in gaming, P2P (KCPTun), and VPN tunnels. Explicitly not a TCP replacement — its README states "sacrifice 10–20% bandwidth in exchange for a transmission speed 1.5×–2× of TCP" on lossy WAN paths.

Source: https://github.com/skywind3000/kcp (commit 1d8a8a4, 2025-02)

Protocol

Wire format — 24-byte header

Offset  Size  Field  Meaning
0       4     conv   Conversation ID (logical connection)
4       1     cmd    Command: PUSH=81, ACK=82, WASK=83, WINS=84
5       1     frg    Fragment index (0 = last/only; N = N more follow)
6       2     wnd    Sender's remaining receive window (in packets)
8       4     ts     Sender timestamp (ms, for RTT measurement)
12      4     sn     Sequence number
16      4     una    Cumulative ACK: all sn < una delivered
20      4     len    Payload length (0 for pure-ACK frames)
24+     var   data   Payload

All little-endian. MTU default 1400 B → MSS = 1376 B. Messages larger than MSS are split into multiple segments with descending frg; all segments must arrive before delivery.

Compare to casting's WalHeader (rsx-cast/src/header.rs):

Offset  Size  Field
0       2     record_type
2       2     reserved (was payload_len)
4       4     crc32          (CRC32C of payload)
8       1     version        (V1 = current; V0 = legacy)
9       7     reserved

16 bytes total, fixed-size #[repr(C, align(64))] payload immediately after, no frg field (casting messages are pre-sized ≤ MTU; the matching engine never produces frames > 256 B).

Reliability model: ACK-based (sender-driven)

KCP detects loss at the sender via absence of ACKs:

1. Every received segment triggers an explicit IKCP_CMD_ACK back to the sender, plus a piggybacked una (cumulative ACK) on every outgoing frame.
2. When the sender sees ACK(N+2), ACK(N+3) but no ACK(N), it increments fastack on segment N.
3. fastack >= resend (turbo: resend=2) → fast retransmit without waiting for RTO.
4. If no ACK arrives within RTO → timeout retransmit.

Contrast with NAK-based (casting, Aeron): the receiver detects the gap on the next datagram with a higher seq and immediately sends NAK(N). The sender retransmits in ~1 RTT.

Latency consequence on zero loss: KCP still sends one ACK per DATA, so every DATA frame triggers a control-plane round-trip. casting on zero loss sends zero control traffic per record — the only background frame is an idle-only heartbeat (RECORD_HEARTBEAT, rsx-cast/src/records.rs) sent every 100 ms when the stream has gone quiet, suppressed entirely while data flows. casting has no flow-control frame; recovery is receiver-driven NAK on a seq gap.

Retransmit horizon

Property	KCP	casting
Source of retransmit	snd_buf (RAM)	hot ring (4 096 slots, RAM) → cold WAL (disk)
Discard condition	Per-segment ACK received	Hot ring is a fixed 4 096-slot LRU; cold tier bounded by WAL retention
Max horizon	Bounded by snd_wnd (default 32, turbo 128)	4 096 hot, 4 h cold (WAL retention)
Survives sender restart	No	Yes (WAL replay)
Audit log	No	Yes (WAL = audit log)

KCP discards a segment from snd_buf as soon as its ACK arrives. A late NAK or a restarted receiver cannot recover any history. casting's cold-tier WAL provides 4 hours of random-access retransmit via read_record_at_seq — long enough for a downstream service to crash, restart, and resume from its last persisted offset.

Flow / congestion control

KCP has two modes: - Standard (nc=0): TCP-style CWND/ssthresh slow-start + congestion avoidance. - Turbo (nc=1): no CWND. Sends as fast as snd_wnd allows. Receiver advertises its window in the wnd header field.

Turbo is correct for an exchange's trusted-LAN use case — a 10 GbE datacenter fabric is not congested and TCP-style CC adds latency without benefit.

casting has no congestion control and no flow-control frame at all (spec §10.4: "Trusted internal network"). Backpressure is handled one layer up: the WAL writer stalls the producer when its flush lag exceeds 10 ms or the buffer fills, and the receiver's bounded reorder buffer bounds how far ahead the sender can run. There is no receiver-advertised window on the wire.

RTO

RFC 6298 SRTT/RTTVAR with KCP's modifications: - Backoff: ×1.5 (vs TCP's ×2) — faster recovery from spurious timeouts. - Min RTO: 30 ms in nodelay mode (nodelay=1), 100 ms otherwise. - Integer millisecond precision throughout. There is no sub-millisecond RTO.

Fastest configuration ("turbo mode")

ikcp_nodelay(kcp, 1, 10, 2, 1);
//               ^  ^   ^  ^
//               |  |   |  nc=1: no CWND
//               |  |   resend=2: fast retransmit after 2 out-of-order ACKs
//               |  interval=10ms: scheduler tick floor
//               nodelay=1: immediate ACK + minRTO=30ms

The interval parameter governs ikcp_update()'s flush cadence. The upstream KCP README recommends 10 ms; the Rust port allows 1 ms. Below 1 ms, ikcp_check() rounds to zero and update() degenerates into a busy spin.

Critical: interval is the floor for the scheduler, not for sends. Calling ikcp_flush() directly after ikcp_send() bypasses the scheduler and writes to the socket immediately (this is what the spin bench measures). The Rust kcp crate also requires at least one update() call before the first flush() (otherwise flush() returns Error::NeedUpdate); the bench pays this once at startup.

Connection model

KCP is connection-less at the wire level. A "connection" is identified by the conv field and is just shared state on both sides — no handshake, no SYN/FIN. The application is responsible for telling KCP the peer's UDP address.

casting is also connection-less (UDP unicast), identified by a matching pair of bind addresses on sender and receiver. Spec §10.4.

Relation to rsx-cast

This is the answer to: "why not just use KCP?"

KCP is an excellent fit for its target problem: low-grade networks (WAN gaming, mobile, P2P) where the underlying RTT is 20–300 ms and a 10× speedup over TCP under loss is competitive. It has no business on an exchange critical path where:

1. The dominant latency is the sendto syscall (~3.85 µs measured locally, see facts/syscall-latency.md), not loss recovery.
2. Every per-DATA ACK doubles control-plane traffic vs casting's NAK-on-gap model.
3. Integer-millisecond RTO is incompatible with a sub-100 µs SLA.
4. No persistence — a producer restart loses all retransmit history; casting's WAL survives.

KCP also fundamentally lacks the audit-log property: every fill, order, and cancel in rsx-cast is on disk before it's on the wire, and the same bytes feed the recorder, the marketdata replay service, and the backtester. KCP would be just a transport.

Guarantees comparison: KCP turbo vs rsx-cast casting

Dimension	KCP turbo (nc=1)	rsx-cast casting
Underlying transport	UDP unicast	UDP unicast
Wire header size	24 B	16 B
Loss detection	Sender (ACK absence + fastack)	Receiver (seq gap → NAK)
Detection latency (zero-loss)	n/a (ACK per DATA always)	n/a (no control plane on success)
Detection latency (1 lost frame)	~2 RTT (need 2 newer ACKs)	~1 RTT (gap seen on next frame)
Retransmit source	snd_buf (RAM, bounded by snd_wnd)	hot ring (4 096) + cold WAL (4 h)
Retransmit horizon	seconds (until ACK arrives)	4 h
Survives sender restart	No	Yes (WAL replay)
Durability	None	WAL = audit log
Min flush granularity	1 ms (Rust port) via timer; immediate via flush()	per sendto (~3.85 µs)
Multi-receiver / fan-out	No (one conv per peer)	Per-receiver via DXS TCP replay; casting itself is unicast
Multiplexed streams	No (single seq space per conv)	No (one stream per CastSender/CastReceiver pair)
FIFO within stream	Yes	Yes
Cross-stream ordering	n/a	n/a (separate WAL files per producer)
Auth / encryption	None	None (trust delegated, spec §10.4)
Congestion control	Optional (nc=0 standard / nc=1 turbo)	None
Zero-loss control-plane overhead	One ACK per DATA	None (idle-only heartbeat every 100 ms)
Heap allocation per send	Yes (snd_buf.push_back)	No (pre-allocated ring slot)
Language ecosystem	C reference + Rust port (kcp 0.6) + Go + many	Rust only (this crate)
Production HFT use	None documented	Target use case
Production gaming use	Extensive (KCPTun, FRP, ~10k stars)	None

Benchmark

benches/compare_all.rs::kcp_spin_flush_128b (run with cargo bench -p rsx-cast --bench compare_all) — Criterion, loopback, 128 B payload (matched to casting's FillRecord, which is mem::size_of::<FillRecord>() == 128). KCP runs under the shared EchoClient harness alongside raw_udp / quinn / tcp.

• kcp_spin_flush_128b — busy-spin server, explicit flush() after every send(). Reveals KCP's true protocol overhead with the scheduler bypassed.

The KCP configuration:

nodelay=1, interval=1ms, resend=2, nc=1, wndsize=128/128, mtu=1400

The earlier standalone compare_kcp.rs (with a naive_1ms timer-driven variant) was folded into compare_all.rs in commit 836cfb1; only the spin-flush variant is retained. The naive 1-ms-timer datapoint was ~11 ms, dominated by the sleep granularity — see "Published numbers" for why that mode is unsuited to an exchange path.

Loss simulation (separate run, requires root):

sudo tc qdisc add dev lo root netem loss 0.1%
cargo bench -p rsx-cast --bench compare_all
sudo tc qdisc del dev lo root

The bench itself does not depend on root or tc.

Note (2026-07-01): compare_all currently aborts on a KCP warmup panic (flush() before update(); bugs.md BENCH-KCP-FLUSH-NEEDUPDATE), so the number below is the last-measured 2026-05-24 figure, not re-run this session.

What this bench is and isn't

This bench measures application-visible loopback RTT using the same Criterion shape, payload size (128 B), and warmup methodology as cast_rtt_bench.rs — making kcp_spin_flush_128b size-comparable to casting's RTT bench (p50 ~9–10 µs on this host; cast_rtt_bench, re-run 2026-07-01).

What it does NOT measure: - Loss recovery (no tc injection in the bench itself). - Multi-stream / fan-out (KCP is single-stream by design). - WAN behaviour (loopback only). - Memory / CPU under sustained load (single-iteration RTT only).

Measured numbers (this host, 2026-05-24)

Bench	p50
cmp_rtt_fill_echo (casting, 128 B)	~9.3 µs
kcp_spin_flush_128b	~17 µs
(removed) naive_1ms_interval	~11 ms (last-measured before folding)

The 17 µs spin number is roughly 1.8× casting — close to the lower bound of KCP's possible per-frame overhead (24 B header parse, ACK list maintenance, Rust port adapter copy). The 11 ms naive number is dominated by the 1 ms sleep granularity on each side.

Published numbers

From the KCP repository's own benchmark wiki (WAN, simulated loss; sender + receiver on separate hosts):

Protocol	Worst-case sample, 10% loss
KCP turbo	195–295 ms
libenet	1 412–1 637 ms

KCP claims a 5–6× advantage over ENet under loss and 1.5×–2× over TCP under "average" loss conditions. These are the headline numbers KCP is known for; they are explicitly WAN/gaming.

Aeron loopback comparison (https://aws.amazon.com/blogs/industries/aeron-on-aws-2025-performance-benchmark-results/, c6in.16xlarge, 100k msg/s): - P50: 21–22 µs - P99: 32–43 µs

casting loopback RTT, this repo: - P50: ~9.3 µs (cast_rtt_bench, re-run 2026-07-01).

KCP and Aeron / casting do not compete in the same latency bracket even on zero-loss loopback.

Where KCP is genuinely better

• Portability: ~1 000 LOC of standards C; ports exist in Go, Rust, Python, Java, JS, Swift, C#. casting is Rust-only.
• Battle-tested on bad networks: gaming and VPN deployments prove KCP works in production with 5–30% loss. casting has never been tested on a public-internet path.
• No persistence requirement: KCP works fine with no disk; rsx-cast casting assumes a WAL.
• Multi-language reach: if you need a client in C# or Swift, KCP wins by existing.

Where casting is genuinely better

• Loopback / LAN latency: ~10 µs RTT vs KCP's ~17 µs spin floor or millisecond timer-driven floor.
• Audit log built in: WAL is the same byte stream as the wire and disk format; one log feeds retransmit, audit, backtesting, and ML training.
• Long retransmit horizon: 4 h via WAL random-access vs bounded by ACK arrival.
• Survives restarts: WAL replay reconstructs sender state exactly; KCP's snd_buf is in-process RAM only.
• Zero control-plane traffic on success: no per-DATA ACK.

Rust ecosystem

Crate	Notes
kcp v0.6	Pure Rust port; sync-friendly; MIT
tokio_kcp	Async stream API on top of kcp
kcp-tokio	Alternative async, claims zero-copy

This bench uses kcp v0.6 (the most direct port of the C reference) to keep the comparison close to the canonical implementation.

Sources

• KCP repo: https://github.com/skywind3000/kcp
• KCP English README: https://github.com/skywind3000/kcp/blob/master/README.en.md
• kcp crate: https://crates.io/crates/kcp
• KCP benchmark wiki: linked from the KCP repo README
• Aeron AWS 2025 numbers: https://aws.amazon.com/blogs/industries/aeron-on-aws-2025-performance-benchmark-results/
• casting local loopback numbers: .ship/18-COMPONENT-BENCHES/LANDSCAPE.md, commit 82e9966 baseline
• Syscall floor: facts/syscall-latency.md