rsx-gateway Architecture

WebSocket gateway process. Accepts client connections, validates orders, routes to Risk via casting/UDP, pushes fills/updates back to clients. Same listening port also serves REST.

Specs: specs/2/11-gateway.md, specs/2/49-webproto.md.

Runtime Model

Single monoio (io_uring) reactor on one thread. All gateway state lives in Rc<RefCell<GatewayState>>; no locks, no cross-thread sharing. Each connection runs as a monoio task spawned from the accept loop. The main loop polls the casting/UDP receiver, ticks the casting sender, sweeps the pending tracker, broadcasts heartbeats, and reaps idle connections.

tokio is NOT used on the gateway hot path. The reference sibling project trader/monoio-client/ proves the same pattern for client-side WS.

Module Layout

REST + WS on One Port

handler.rs::handle_connection reads the initial HTTP request once. ws::is_ws_upgrade decides:

• WS upgrade -> ws_handshake_from_request (JWT extract + 101 Switching Protocols) -> per-frame loop
• Otherwise -> rest::handle_rest writes the response and closes the connection

/health and /v1/symbols are the only REST surfaces today.

JWT Authentication

jwt::validate_jwt_with_claims enforces all of:

• HS256 signature against RSX_GW_JWT_SECRET
• exp (expiry)
• nbf (not-before, when present) -- validate_nbf = true
• aud == "rsx-gateway"
• iss == "rsx-auth"

Boot refuses to start if RSX_GW_JWT_SECRET is shorter than JWT_SECRET_MIN_LEN = 32 bytes (HS256 floor). Empty secret also exits 2.

Claims carries a jti. JtiTracker (FIFO-bounded, 16 K-entry cap) is wired on the WS handshake path: ws.rs holds a process-wide JTI_TRACKER and validates JWT + claims + rejects replay via extract_user_and_record_jti — a token missing jti is rejected, and a replayed jti is refused (401). Covered by tests/jwt_ws_e2e_test.rs. Replay state is per-process; a shared (Redis) tracker for multi- gateway deployments is a future option, not needed for the single- gateway topology today.

Rate Limiting

Two layers checked in sequence on every NewOrder frame:

Either exhausted -> WS error code 1006 "rate limited", no queueing.

The per-IP limiter map is bounded at IP_LIMITER_MAX = 10_000 entries. A parallel VecDeque<IpAddr> records insertion order; on overflow the oldest IP is evicted FIFO. This prevents memory exhaustion from a rotating-IP adversary while preserving normal-case fairness. Covered by tests/state_test.rs::ip_limiter_map_is_bounded.

rate_limit::RateLimiter uses microsecond-resolution token accounting (capacity * 1_000_000 internal units) for fair sub-second refill.

Circuit Breaker

state.rs::CircuitBreaker is fail-CLOSED on overload. States: Closed -> Open on threshold consecutive failures; Open -> HalfOpen after cooldown; HalfOpen allows one probe; success -> Closed, failure -> back to Open.

When circuit.allow() returns false, the gateway responds WS error code 5 "overloaded" and refuses to forward the order to Risk. Defaults: threshold=10, cooldown=30s.

Connection Limits

Hard cap of 5 concurrent connections per user_id (state.rs::add_connection). Sixth attempt is rejected at handshake time.

Frame Size Cap

Both ws::ws_read_frame and ws::ws_read_frame_buf reject WS payloads larger than 4096 bytes with InvalidData. No fragmentation support; orders and cancels comfortably fit.

Message Flow

Client (WS JSON)
    |
    v
+-- handler.rs ----------------------------------+
|  1. Wait readable (10ms timeout for poll)      |
|  2. Read frame (ws::ws_read_frame_buf, 4KB)    |
|  3. UTF-8 + protocol::parse                    |
|     NewOrder:                                   |
|  4. Per-IP + per-user limiter                  |
|  5. Circuit breaker `allow()`                  |
|  6. Symbol bound + tick/lot alignment          |
|  7. UUIDv7 oid; pending.push                   |
|  8. CastSender::send_raw(ORDER_REQUEST, &bytes) |
|     (binary forward path; alloc-free)          |
+-------------------------------------------------+
    |
    v [casting/UDP -> Risk]
  Risk -> Matching -> Risk
    |
    v [casting/UDP -> Gateway]
+-- main.rs casting loop ----------------------------+
|  Decode FillRecord/OrderInsertedRecord/...     |
|  Dispatch to route.rs                          |
+-- route.rs ------------------------------------+
|  Serialize WsFrame JSON, push_to_user          |
+-------------------------------------------------+
    |
    v
+-- handler.rs ----------------------------------+
|  drain_outbound -> ws_write_text (JSON)        |
+-------------------------------------------------+

The binary forward path (OrderRequest and CancelRequest written via send_raw from a stack struct) is allocation- free on the hot path. The outbound JSON fan-out path allocates per frame (per-connection VecDeque<String>); acceptable for the WS-JSON protocol per spec.

Pending Order Tracking

PendingOrders (capacity = RSX_GW_MAX_PENDING, default 10k) tracks every order sent to Risk until a terminal OrderDone/OrderCancelled/OrderFailed arrives. Stale entries past RSX_GW_ORDER_TIMEOUT_MS (default 10s) are swept every 100ms by the main loop. Push failure when full returns WS error 1003 "pending queue full".

Heartbeats

Server sends WS heartbeat every RSX_GW_HEARTBEAT_INTERVAL_S (default 5s). Per-connection idle reaper closes connections with no activity for RSX_GW_IDLE_TIMEOUT_S (default 10s). Per-connection heartbeat timeout is also enforced inside handle_connection.

Backpressure & Failure Modes

No internal queueing -- always fail fast.

Statelessness

Gateway holds no durable state. On crash: - User sessions drop, in-flight orders may be lost - Recovery: clients reconnect; Risk + WAL preserve authoritative state - Fills are persisted by Risk/Recorder, never gated by Gateway

Scaling

Horizontal by user_id hash. Load balancer routes sticky sessions. Each gateway connects to its Risk shard via casting/UDP. No cross-instance coordination.

Networking (monoio / io_uring)

Gateway uses monoio (io_uring) for all client-facing I/O: - io_uring batches submissions in shared kernel/userspace rings -- fewer syscalls than epoll - For 100K+ connections, epoll syscall overhead is too high - Future: DPDK/AF_XDP swaps the I/O layer without touching the connection model

Architectural Decisions

Runtime: monoio (io_uring). The gateway sits on the GW→ME→GW critical path and multiplexes many concurrent WS clients. Every epoll syscall is a tail-latency contributor; io_uring batches submissions in shared kernel/userspace rings and pays a fraction of the per-event cost. The single-threaded monoio reactor keeps gateway state in Rc<RefCell<_>> — no locks, no cross-thread hand-off.

The gateway is NOT a tile in the strict sense: the inner loop is select! over many sockets, not a busy-spin on one SPSC ring. Tiles are for compute-bound stages; monoio is for I/O-multiplex-bound stages. See ../notes/tiles.md for when each is appropriate.

20260530 — real-gateway WS + REST order latency (warmed clients, baseline)

What: order round-trip driven through the real rsx-gateway over its actual WebSocket + REST transport with warmed clients, against a live cluster (gateway + risk + matching; mark/marketdata/recorder skipped — not on the resting-order path). Postgres in docker, 110 funded users seeded. Hand-rolled blocking WS client (no async runtime), one OS thread per connection, JWT via jsonwebtoken. Closed-loop (one in-flight order per stream).

Numbers (all pairing-OK, 0 transport errors, ME survived)

Orders rested (non-crossing GTC limit BUYs; no maker, so none filled). The client floor (15 µs p50) ran the identical masked-send / unmasked-read path over loopback echo and is ~subtractable from the gateway numbers.

Conclusion

The ~11.5 ms WS p50 is not transport and not the client: REST over the same gateway is 131 µs and the client floor is 15 µs. It confirms the known root cause (see 20260530_e2e-ws-probe.md, GATEWAY-LATENCY in bugs.md): the gateway's single monoio reactor parks the per-connection handler in a 10 ms readable() timeout while the casting-recv loop delivers the response — poll-loop egress starvation, internal reactor sharing, not the wire. This is the baseline for the planned gateway egress-tile-split (decouple casting-recv to a pinned busy-spin tile → SPSC → WS writers; shard reactors).

Caveats

• Closed-loop (one in-flight order per stream) — not a saturation / coordinated-omission test; the rate column is descriptive, not a throughput ceiling.
• Client threads float on non-pinned cores 0/5 while gw/risk/ME are pinned 1–3 (scheduler noise possible).
• Run against a fresh, empty ME book.
• Why single stream is 30k, not 100k: with no maker and IOC not honored (see IOC-NOT-HONORED, bugs.md), every order rests and uncancelled GTCs exhaust rsx-book's fixed 65,536-deep slab (it panicked the ME mid-build). The harness has a slab-budget guard refusing to run if single_n + warmup + par_conn*(par_n+par_warmup) ≥ 65536; defaults sized to 61k < 65k. 100k uncancelled resting orders on one ME without a maker/IOC is physically impossible.

Source: rsx-gateway/benches/ws_order_latency.rs (commit landing this report), live playground start-all cluster, 6-core box.