rsx-matching Architecture

A matching engine is the part of an exchange that pairs orders. An incoming buy that meets a resting sell (or a sell that meets a resting buy) becomes a fill — a trade; anything that doesn't cross rests in the book and waits. This process does exactly that for one symbol: it takes orders from Risk, matches them against the resting orderbook, decides the fills, records them, and tells Risk and Marketdata what happened.

The match is price-agnostic — it pairs orders on their own limit prices alone, with no notion of mark or index price (those live in Risk and Mark). And it is flat: the 2026-07-03 bench measures the match at ~30 ns whether the book holds one resting order or 100 000 (depth-independent — see Measured Performance below). Scale-out is by symbol: one matching-engine instance per tradeable symbol, added independently of user-shard (Risk) scale-out.

Matching is the authoritative writer of fills, accepts, cancels, order-failed, and config-applied records to the WAL: once it persists a fill, that fill happened.

Specs: specs/2/17-matching.md, specs/2/45-tiles.md §3.1.

Trust Boundary

The matching engine does not validate user input — by design. By the time an order reaches this process it has already passed two upstream gates: the gateway authenticates the client (JWT, TLS) and checks structural well-formedness, and the risk tile checks margin and position (rsx-risk, the pre-trade authority). ME assumes its inputs are well-formed and in-shard, and does not re-validate on the hot path. This is the single-owner rule from the repo trust-boundary policy ("The matching engine doesn't validate user input" — CLAUDE.md; specs/2/47-validation-edge-cases.md owns where each check lives). Adding re-validation here would duplicate an upstream owner and slow the hot loop for no correctness gain.

The one thing ME is strict about is its own output: every WAL append on the fill path crashes on failure rather than silently dropping a record (see WAL Append below), because ME is the authoritative writer and a lost fill is unrecoverable.

Measured Performance

In-process microbenchmarks, single box, no UDP/WS — compute floors, not wire-to-wire latencies. Captured 2026-07-03 (reports/20260703_matching-benches.md, cargo bench -p rsx-matching, p50 over 50 samples, timed thread pinned to core 2).

Point	p50	What
match_by_depth/n=1	~30 ns	the match itself
match_by_depth/n=100000	~30 ns	same at 100k resting — depth-independent
dedup/hit_duplicate	3.7 ns	duplicate order rejected
dedup/insert_new	147 ns	FxHashMap insert
wal_events/append_1_fill	84 ns	serialize 1 fill (no fsync)
me_accept_path/full	266 ns	full accept: dedup + WAL accept + match + WAL events + index, 1 fill
me_throughput/orders	281 ns	≈ 3.6M orders/s (1 fill each)
wal_replay_30k_records	32.8 ms	≈ 915k records/s cold replay

The order-type and multi-level-sweep figures in that report are quarantined (a 10k-level fixture artifact, flagged for a Phase-2 audit) — do not cite them as per-order-type latency. The single-op and full-accept figures above are the trusted set. These are indicative (shared 4-core docker host, cluster stopped); re-run on a quiet box for a citable baseline. The full GW→ME→GW round-trip is transport-bound (~4 casting hops), not compute-bound.

Module Layout

File	Purpose
main.rs	Binary: casting setup, WAL init, match loop, event routing, cancel index
wire.rs	OrderMessage — #[repr(C)] casting wire type for inbound orders
dedup.rs	DedupTracker — 5-minute sliding-window duplicate detection
config.rs	poll_scheduled_configs(), write_applied_config() — Postgres config polling
wal.rs	publish_events(), flush_if_due(), write_events_to_wal() (replay + bench helper)

Execution: a single pinned loop

Matching has no tile structure — it is one core-pinned thread running a loop, with no SPSC rings (what specs/2/45-tiles.md §3.1 calls a "degenerate tile"; with no rings between tiles, it is just a loop). All work — casting I/O, dedup, matching, WAL append, casting fan-out — happens inline on the pinned core. No intra-process IPC, no cross-thread queues on the hot path.

Pinning: RSX_ME_CORE_ID selects the core (core_affinity::set_for_current, main.rs:195-200).

Main Loop

Tight busy-spin on the pinned core (main.rs:403):

loop {
    1. cast_receiver.try_recv()       // OrderRequest or CancelRequest from risk
    2. dedup.check_and_insert()       // 5-minute sliding window
    3. wal.append_framed(ORDER_ACCEPTED) // authoritative — panic on err
    4. process_new_order(book)        // match against book; events in fixed buffer
    5. publish_events()               // one-CRC fan-out: WAL + cast(risk) + cast(mkt)
    6. update_order_index(events)     // O(1) cancel index maintenance
    7. flush_if_due(wal, 10ms)        // periodic WAL flush
    8. poll_scheduled_configs()       // every 10 minutes (Postgres)
}

Cancels share the loop: process_cancel() consults order_index for O(1) slab lookup, then runs steps 4-6 via the same publish_events fan-out.

O(1) Cancel Index

type OrderKey = (u32, u64, u64); // (user_id, oid_hi, oid_lo) FxHashMap<OrderKey, slab_handle: u32> rebuilt incrementally from book.events() after every match and cancel cycle (update_order_index, main.rs:59).

• OrderInserted → insert (key, handle)
• OrderDone → remove key (fires on fully-filled or cancelled, so the index never leaks)

Replaces the prior linear slab scan over a 65 536-slot arena. A defensive check inside process_cancel still verifies the slab slot matches (user, oid) after the index hit; mismatch warns and bails without crashing.

Commit: cdc9360.

WAL Append: Crash on Failure

Matching is the authoritative writer for the fill path. Every WAL append uses .expect(...) with a named-invariant message (commit 82a9206):

• ORDER_ACCEPTED — "violates 6-consistency.md invariant 7 (WAL persistence) and breaks dedup on replay"
• Event path (Fill / OrderInserted / OrderDone) — "violates 6-consistency.md invariant 1 (totally-ordered events) and ordering rule 'Fills precede ORDER_DONE'"
• CANCEL — "violates invariant 1 and invariant 5 (ORDER_DONE commit boundary)"
• ORDER_FAILED (duplicate-reject) — "violates invariant 7"
• CONFIG_APPLIED — "violates invariant 7; CONFIG_APPLIED must precede casting fan-out"

Design choice: matching engine is authoritative; silently losing a fill violates Invariants #1 and #2. Crash, let the replica take over, replay from WAL tip. casting fan-out sends remain best-effort (receivers recover via NAK / TCP replay) and only warn on failure.

Event Fanout

Fixed array [Event; MAX_EVENTS] (MAX_EVENTS = 65_536, heap-boxed) on the orderbook struct, reset per match cycle. Two independent CastSenders:

• ME → Risk: fills, BBO, order done/failed (all events)
• ME → Marketdata: inserts, cancels, fills

publish_events (wal.rs) prepares each record once (single CRC + seq) and fans the resulting Framed to WAL + cast + (optionally) mkt with send_framed / append_framed — no re-CRC per destination. Routing per event type:

Event	WAL	cast (risk)	mkt
Fill / OrderInserted / OrderCancelled	yes	yes	yes
OrderDone	yes	yes	no
OrderFailed	yes	no	no
BBO	no (derived on replay)	yes	yes

BBO is the one event not framed by the WAL; both senders use their own seq counter via CastSender::send for that record.

Deduplication

DedupTracker keeps (user_id, oid_hi, oid_lo) for a 5-minute sliding window. On replay, the dedup set is rebuilt from RECORD_ORDER_ACCEPTED records in the WAL — duplicate detection is WAL-persisted, not a memory-only guard.

Config Hot Reload

poll_scheduled_configs() queries symbol_config_schedule every 10 minutes (main.rs:559, 600 seconds). When a new version is due, the matcher writes CONFIG_APPLIED to WAL before fanning out to casting, then applies tick/lot changes live. On startup, the current version emits one CONFIG_APPLIED record too.

Architectural Decisions

Runtime: a single pinned loop + tokio sidecar. The matching loop is the lowest-latency stage of the system — ~266 ns p50 for the full accept path (dedup + WAL accept + match + WAL events + order-index update), per the 2026-07-03 bench (me_accept_path/full; see Measured Performance). Network I/O multiplexing does not appear in the inner loop; the only sockets are one CastReceiver (orders in) and two CastSenders (events to risk and marketdata). With nothing to multiplex, the cheapest reactor is no reactor: one pinned thread, busy-spin, all work inline on the cache-warm core — a loop, not a tile (the "degenerate tile" of ../specs/2/45-tiles.md §3.1: no SPSC rings, so no tile structure).

A tokio sidecar handles the cold path — poll_scheduled_configs() queries Postgres every 10 minutes for symbol config updates. That is blocking I/O and explicitly off the hot loop. See ../notes/tiles.md for the broader pattern.

Cold-start snapshot + WAL replay (wal.rs)

Snapshots are written every 10 s with atomic rename (snapshot.bin + wal_seq.txt sidecar). Between snapshots, SIGKILL would lose every fill — recovery replays the WAL from sidecar + 1:

• RECORD_ORDER_ACCEPTED → re-runs process_new_order to deterministically regenerate fills + side-effect events.
• RECORD_ORDER_CANCELLED → book.cancel_order(handle) against the reconstructed order_index.
• Other record types (Fill, OrderInserted, OrderDone, BBO) are skipped — they are side effects of accepted-order replay.

replay_wal_after_snapshot returns the highest WAL seq applied; caller seeds WalWriter::next_seq = ret + 1 so subsequent live writes never reuse a replayed seq.

FAULTED → skip-the-gap (order stream is drop-safe)

When CastReceiver::try_recv returns CastRecv::Faulted (a gap that outran in-band NAK recovery), the matching loop skips the gap and resumes live — it does NOT replay and does NOT panic. On FAULTED it counts the skipped seqs (gauges.drops), warns with the gap range, then calls cast_receiver.reset_after_replay( gap_end_inclusive) to resume live UDP delivery from gap_end_inclusive + 1.

This is sound because the risk→ME order stream is recovered at the application layer, not the transport:

• A dropped pre-ack order is re-sent by the client (no-ack-within- timeout, specs/2/49-webproto.md) and deduped on the ME's WAL (RECORD_ORDER_ACCEPTED) — exactly-once.
• The ME re-sequences on output (its own WAL seq), so an inbound gap is never an output gap: risk / recorder / marketdata still see a contiguous ME stream.

Fill delivery in the other direction (ME→risk) is what genuinely needs recovery, and it has its own path: the ME runs a WAL replication server (RSX_ME_REPLICATION_BIND_ADDR) that risk pulls from on risk-side FAULTED. ME cold-start recovery replays the ME's own local WAL (snapshot + replay_wal_after_snapshot); neither depends on a remote order-replay consumer, which is why the old RSX_ME_REPLICATION_ADDR pull path was removed.

Live ingestion samples me_in / me_dedup_done / me_wal_* / me_match_done on the hot path.

rsx-matching benches — uniform harness baseline (Phase 1)

Date: 2026-07-03 Crate: rsx-matching Sprint: .ship/31-BOOK-MATCH-CAST-TREATMENT Phase 1 (Measure) Status: harness + bench set + numbers captured (cluster off; indicative on a shared 4-core docker host). match_by_depth / dedup / WAL / accept are trusted; the order-type + sweep µs figures are measured but UNRECONCILED (fixture artifact — see Numbers) and flagged for the Phase-2 codex audit.

What this is

Phase 1 of the "cast treatment" for the matching engine: consolidate the four existing rsx-matching benches onto ONE shared harness so every figure is measured against identically-constructed state, with the same core pinning and Criterion statistics — a clean, directly-comparable set of rsx-matching's OWN numbers. No competitor baselines yet (that is Phase 2).

All numbers are single-box, in-process microbenchmarks — the matching algorithm and its accept path in isolation, no UDP / WS / cross-process transport. They are compute floors, not wire-to-wire latencies. Run it yourself: cargo bench -p rsx-matching.

The uniform harness (rsx-matching/benches/harness.rs)

One module, included by every bench via #[path = "harness.rs"] mod harness;. It fixes the things that, if they drifted per-bench, would make the numbers unfair:

• Core pinning: the timed Criterion thread pins to core 2 (harness::pin(), core_affinity) — same convention as the cast and book harnesses, so cross-crate runs share the core.
• Criterion config: harness::criterion() = sample_size(50) — matches the cast/book convention so statistics are comparable.
• Symbol config: tick 1, lot 1 → raw fixed-point units; MID = 100_000 (same mid the prior matching benches used, so carried-over numbers line up).
• Shared fixtures:
• build_book(depth) — a book of depth resting asks laddered up from MID+1, best level BIG_QTY (1e9). A qty-1 taker at MID+1 does one non-draining partial fill, so the match work is held constant while only resting depth varies. Deterministic.
• single_ask(qty) — a one-level book for the by-order-type benches.
• Me — the full ME critical section as a reusable fixture (real Orderbook seeded to a depth + real WalWriter on a tempdir + real DedupTracker + real FxHashMap order index). Me::accept() runs the exact sequence the ME main loop runs between me_in and me_out (sans cast send): dedup check → OrderAcceptedRecord WAL append → process_new_order → write_events_to_wal → order-index update.

No bench re-rolls its own config, pin, or symbol — drift is how unfair numbers creep in.

The bench set

Six Criterion groups across six bench files, each measuring one concern.

Group	File	What it measures
match_by_depth/n={1,100,1k,10k,100k}	match_depth_bench.rs	One qty-1 taker fill vs resting book depth. Match work held constant; isolates whether a fuller book/slab slows a single match (should be O(1) best-level access, depth-independent).
match_by_order_type/{gtc_full_cross,ioc,fok,post_only_rest,reduce_only}	match_by_type_bench.rs	Cost of each order type's distinct path: full cross, IOC residual-done, FOK liquidity check + fill, post-only that rests, reduce-only sell reducing a real long position against a resting bid.
sweep_n_levels/n={1,5,20,100}	match_n_levels_bench.rs	One aggressor sweeping N single-order price levels (partial fills across levels) — how the match loop scales with fills (O(consumed)).
dedup/{insert_new,hit_duplicate,cleanup_10k}	matching_bench.rs	The duplicate-order guard every accepted order pays (FxHashMap insert / hit / bulk 10k cleanup).
wal_events/{append_1_fill,drain_10_fills,drain_100_fills}	matching_bench.rs	Serializing a match's emitted events to WAL (write_events_to_wal, no fsync) + draining the event buffer the risk/mkt fan-out iterates.
me_accept_path/full, me_throughput/orders	process_order_bench.rs	The full Me::accept() critical section — per-order latency (p50) and orders/s. Each accept does one fill, so fills/s == orders/s here.
wal_replay_30k_records	wal_replay_bench.rs	Cold-start WAL replay: drain 30k records (10k accepted + 10k fill + 10k bbo) — the cost an ME restart pays before its first order.

Pure orderbook data-structure micro-benches (slab alloc/free, price→index compression) were dropped from the matching set — they belong to rsx-book's bench set and would double-count here.

Numbers

Captured 2026-07-03, RSX cluster STOPPED (busy-spin tiles off), sample_size 50, timed thread pinned to core 2. The box is a shared 4-core docker host with residual docker load, so these are indicative p50s (robust over 50 samples), not an isolated-box baseline — re-run on a quiet box for a citable baseline. Grouped by how much I trust each figure.

Trusted (clean, single-op or well-scoped)

Point	p50	Note
match_by_depth/n=1	30.4 ns	match algorithm only
match_by_depth/n=100	30.8 ns
match_by_depth/n=1000	29.3 ns
match_by_depth/n=10000	32.7 ns
match_by_depth/n=100000	29.7 ns	depth-INDEPENDENT
me_accept_path/full	266 ns	full Me::accept (dedup+match+buffered WAL+index), 1 fill
me_throughput/orders	281 ns	≈ 3.6M orders/s (1 fill each)
dedup/insert_new	147 ns	FxHashMap insert
dedup/hit_duplicate	3.7 ns	duplicate rejected
dedup/cleanup_10k	522 µs	bulk 10k prune
wal_events/append_1_fill	84 ns	serialize 1 fill (no fsync)
wal_events/drain_10_fills	518 ns
wal_events/drain_100_fills	556 ns
wal_replay_30k_records	32.8 ms	≈ 915k records/s cold replay

Headline: the match itself is ~30 ns, flat across depth 1→100k (depth-independent, consistent with the 52 ns deep-book figure in 20260530_component-benches.md); a full single-order accept is 266 ns; duplicate rejection is 3.7 ns.

Measured but UNRECONCILED — do NOT cite as order-type latency

Point	p50 (raw)
match_by_order_type/gtc_full_cross	120.7 µs
match_by_order_type/ioc	32.3 µs
match_by_order_type/fok	73.1 µs
match_by_order_type/post_only_rest	69.3 µs
match_by_order_type/reduce_only	64.8 µs
sweep_n_levels/n=1	34.5 µs
sweep_n_levels/n=5	38.3 µs
sweep_n_levels/n=20	147 µs
sweep_n_levels/n=100	578 µs

ANOMALY — flagged for the Phase-2 codex faithfulness audit. post_only_rest crosses nothing yet measures 69 µs — 260× the 266 ns single-accept and 2000× the 30 ns match. The order-type/sweep benches use an iter_batched depth-10k book fixture; the µs scale is inconsistent with the match/accept floors and most likely reflects the 10k-level fixture's allocation/drop cost bleeding into the timed region (or, less likely, a real O(depth) cost in the accept path — which would itself be a finding). Either way these numbers are NOT the per-order-type dispatch cost. Needs a shallow-book (or explicit-drop-excluded) rerun + codex review before use; sweep_n_levels scaling (34→578 µs for 1→100 levels) is directionally right (O(levels consumed)) but carries the same fixture caveat.

Caveats (honesty guardrails)

• Single box, in-process microbench. No UDP/WS/cross-process. These are compute floors; the full GW→ME→GW round-trip is transport-bound (~4 casting hops), not compute-bound — see 20260530_e2e-ws-probe.md.
• Indicative, not isolated-baseline. Captured with the RSX cluster stopped (busy-spin tiles off cores 2/3), but on a shared 4-core docker host with residual load. p50 over 50 samples is robust for the trusted single-op figures; the order-type/sweep µs figures are quarantined for a different reason (fixture artifact, see Numbers), not host noise. Re-run on a quiet box for a citable baseline.
• Reproduce: cargo bench -p rsx-matching (all groups) or per file, e.g. cargo bench -p rsx-matching --bench match_depth_bench. Pins to core 2; ensure no busy-spin tile is pinned there.
• Fixture design charges every point equally: same harness, same core, same Criterion config, same non-draining best level for the depth and accept-path benches, so depth is the only variable in match_by_depth.
• fills/s == orders/s in me_throughput only because each accept does exactly one fill by construction; not a claim about multi-fill sweeps (see sweep_n_levels).

Next (Phase 2)

Competitor baselines under a shared compare-harness (LMAX Disruptor-style matcher, liquibook if feasible, naive matcher) — rsx-matching/compare/ with the [lib]/[reimpl]/[pub] provenance taxonomy. Not started.