Architecture

Zion is a single async Rust binary built on Tokio, Hyper, and rustls. ~15,900 lines across 21 source files.

Module map

src/
├── main.rs        # Entry point and `async_main`: the boot steps in order, then wait for the shutdown signal
├── boot.rs        # The boot steps themselves (config, TLS, state, admin API, background tasks, listeners, drain)
├── accept.rs      # Accept loops, the per-connection HTTPS service (mTLS/JA4 identity headers, health fast path) and the :80 handler
├── state.rs       # AppState, ResolvedAppConfig (ArcSwap snapshot) and the per-source limiters shared by every handler
├── http_util.rs   # Request-ID generation and the small response builders (empty/text/405/401, security headers)
├── dispatch.rs    # Request pipeline: routing (LRU + radix), WAF gates, cache, CORS, metrics; thread-local route LRU lives here
├── dispatch/      # The pipeline's stages as modules: `gates` (pre-routing, CORS, auth), `route` (route lookup, upstream choice), `waf_gate` (URI, headers, body), `cached` (cached routes: lookup, singleflight, revalidation, store), `telemetry` (trace context, access log)
├── config.rs      # TOML schema (serde types), parsing and validation
├── routing.rs     # Resolves the parsed routes into ResolvedRoute + the host-aware radix HostRouter
├── tls.rs         # TLS config, SNI resolution, session tickets, 0-RTT, hot-reload, predictive prewarming
├── waf.rs         # 5-gate WAF pipeline (Aho-Corasick balanced/aggressive, entropy, simd-json)
├── proxy.rs       # Upstream forwarding + XffMode policy (append/rewrite/drop), WebSocket TLS-to-upstream
├── cache.rs       # Two-level cache: L1 thread-local (O(1) LRU) + L2 DashMap
├── security.rs    # CORS (FNV O(1)), rate limiter (packed-atomic, lock-free), security headers, trusted-proxy CIDR resolution
├── metrics.rs     # Prometheus: sharded counters, differential histogram, ArcSwap render
├── health.rs      # Upstream health checker, EWMA latency, gray failure detection
├── pool.rs        # Pool member choice (power of two choices on in-flight × peak-EWMA latency) and passive outlier ejection
├── breaker.rs     # Per-upstream circuit breaker (sliding window, half-open probe token, injectable clock)
├── bulkhead.rs    # Per-upstream `max_in_flight` cap: counter registry + a body wrapper that holds the slot until the response is sent
├── dns.rs         # Upstream DNS with stale-on-error: last good answer per host, one lookup in flight per host, deadline
├── drain.rs       # Graceful drain on SIGTERM: idle keep-alive closed, HTTP/2 GOAWAY, requests in flight finished
├── logq.rs        # Bounded lossy queue in front of stderr, so a stalled log pipe never stalls a request
├── vary.rs        # `Vary` secondary cache keys (RFC 9111 §4.1), bounded per primary key
├── http_conditional.rs  # `If-None-Match` / `If-Modified-Since` evaluation (RFC 9110 §13)
├── static_files.rs # `mode = "static"` file server (ETag, Range, precompressed sidecars) and the shared `Range` parser
├── uri_norm.rs    # Request-path normalisation before routing (RFC 3986 §6.2.2)
├── via.rs         # `Via` header and loop detection (RFC 9110 §7.6.3)
├── reload.rs      # Config hot-reload: rebuild the snapshot, carry health/breaker/pool state across it
├── audit.rs       # HMAC-chained audit log, PII redaction (`[redact]`: headers, query params, client IP)
├── observability.rs # W3C Trace Context, panic hook, audit/trace counters
├── connlimit.rs   # Per-IP concurrent-connection cap (enforced at accept)
├── tarpit.rs      # Tarpit for flooding sources
├── numa.rs        # NUMA-aware shard wrapper around DashMap (feature: --features numa-aware)
├── atomic_file.rs # Crash-safe file replace for certs/keys and persisted config pushes
├── admin.rs       # Admin API (config push, JWT revocation); internal IPs or mTLS, write token
├── auth.rs        # JWT/OIDC validation gate (feature: --features auth)
├── acme.rs        # ACME HTTP-01 auto-renewal (feature: --features acme)
├── quic.rs        # HTTP/3 QUIC listener (feature: --features http3)
├── listener.rs    # Listener supervisor: re-binds :80/:443 when `[server.listen_*]` changes in a hot reload
├── uring.rs       # io_uring single-shot accept (feature: --features io-uring-accept)
├── bootstrap.rs   # Platform detection (CPU, RAM, L1d cache, AES-NI/NEON, kernel features) + AES-GCM calibration
├── net.rs         # Socket tuning (SO_REUSEPORT, TCP_FASTOPEN, TCP_QUICKACK, TCP_DEFER_ACCEPT, TCP_CORK, SO_BUSY_POLL)
├── doctor.rs      # `zion doctor` environment diagnostic
├── init.rs        # `zion init` interactive wizard + `zion auto` ephemeral mode (feature: --features init)
├── tui.rs         # `zion top` live dashboard (feature: --features tui)
├── cli.rs         # Command-line argument parsing
└── logging.rs     # Structured logging (text/JSON)

Request lifecycle

Client
  │
  ├─ HTTP :80 ──────► ACME challenge proxy OR 301 → HTTPS
  │
  ├─ HTTPS :443 (TCP) ──► TLS 1.3 (HTTP/1.1 + HTTP/2)
  │
  └─ HTTPS :443 (UDP) ──► QUIC (HTTP/3, feature-gated)
       │
       ▼
  ┌─ TLS Handshake ─────────────────────────────────────┐
  │  rustls + SNI resolution (SingleCert or SniResolver) │
  │  Session cache (16384 entries), 0-RTT early data     │
  │  Hardware crypto: AES-NI (x86), AES-CE (ARM)         │
  │  mTLS (optional): SHA-256 leaf fingerprint extracted │
  │     and forwarded as `X-Client-Cert-Fingerprint`     │
  └──────────────────────────────────────────────────────┘
       │
       ▼
  ┌─ Health Probe Fast-Path ───────────────────────────┐
  │  /healthz, /readyz → 200 OK (~1us, bypasses below) │
  └────────────────────────────────────────────────────┘
       │
       ▼
  ┌─ Pre-routing Security Gates (order = PRE_ROUTING) ────┐
  │  1. Scrub client-supplied identity headers             │
  │  2. Scrub client routing/host override headers         │
  │  3. URI length check (path+query >8192 bytes → 414)   │
  │  4. Path normalisation (RFC 3986 §6.2.2)               │
  │  5. Method whitelist (7 methods, else 405)             │
  │  6. 0-RTT replay protection (non-idempotent → 425)    │
  │  7. Loop detection (Via, → 508)                        │
  │  8. Per-IP rate limiter (packed atomic, lock-free)     │
  │  9. Feature-gated: sovereign class, TLS fingerprint,   │
  │     mesh score                                         │
  │  then: built-in endpoints (/metrics → internal IPs),   │
  │        CORS pre-flight (OPTIONS → 204)                 │
  └────────────────────────────────────────────────────────┘
       │
       ▼
  ┌─ Route Lookup ─────────────────────────────────────┐
  │  Thread-local LRU (FNV hash, O(1), cap 256) →      │
  │    fall through to matchit radix tree on miss →    │
  │    promote to MRU. Returns ResolvedRoute (pre-     │
  │    parsed URI, upstream, WAF profile, cache cfg).  │
  └────────────────────────────────────────────────────┘
       │
       ▼
  ┌─ Auth Gate (if --features auth + auth_profile set) ┐
  │  JWT validation: HMAC/RSA/ECDSA via jsonwebtoken   │
  │  Claims forwarded as X-Auth-Subject, X-Auth-Email  │
  └────────────────────────────────────────────────────┘
       │
       ▼
  ┌─ WAF Pipeline (if enabled on route) ───────────────┐
  │  Gate 1: Body size enforcement (O(1))               │
  │  Gate 2: Content-Type validation (delimiter-aware)  │
  │  Gate 3: Aho-Corasick scan — pattern set selected   │
  │           per profile mode (balanced ~100 /         │
  │           aggressive ~240); raw + iterative         │
  │           normalisation (URL-decode, SQL comment    │
  │           strip, JSON unicode), up to 3 passes      │
  │  Gate 4: Entropy analysis (Shannon ≥256 bytes;      │
  │           JSON-string-only for application/json;    │
  │           per-profile threshold + kill-switch)      │
  │  Gate 5: JSON structural validation (simd-json) +   │
  │           depth + per-string length limits          │
  └────────────────────────────────────────────────────┘
       │
       ▼
  ┌─ Upstream protection (opt-in, per upstream) ───────┐
  │  max_in_flight  → bulkhead: full → 503 at once      │
  │  circuit_breaker → open → 503 + Retry-After         │
  │  pool pick      → P2C on in-flight × peak-EWMA;     │
  │                   skips down / gray / ejected       │
  └────────────────────────────────────────────────────┘
       │
       ▼
  ┌─ Dispatch by Mode ─────────────────────────────────┐
  │  standard     → proxy_pass (pooled HTTP client)    │
  │  sse_stream   → proxy_pass_stream (no buffering)   │
  │  static_cache → singleflight (watch channel) +     │
  │                  L1/L2 lookup/fetch + tee-reader   │
  │                  (cache build runs in parallel     │
  │                  with client streaming)            │
  │  websocket    → HTTP Upgrade + bidirectional pipe  │
  │                  (TLS-to-upstream supported)       │
  │                                                     │
  │  All modes apply X-Forwarded-For policy:           │
  │    append   → preserve inbound chain + add our IP  │
  │    rewrite  → drop inbound, emit single trusted    │
  │    drop     → strip, emit nothing                  │
  └────────────────────────────────────────────────────┘
       │
       ▼
  ┌─ Response Processing ──────────────────────────────┐
  │  Security headers (HSTS, XFO, XCTO, Referrer,      │
  │    Permissions-Policy, hop-by-hop stripping)       │
  │  CORS headers (if origin matched, FNV O(1))        │
  │  X-Request-ID (stack-buffer, zero alloc)           │
  │  W3C traceparent (stack-buffer, zero alloc)        │
  │  Alt-Svc: h3 (if --features http3)                 │
  │  Prometheus metrics (sharded counters, ~2ns)       │
  └────────────────────────────────────────────────────┘
       │
       ▼
  Client

Design decisions

Decision	Rationale
mimalloc global allocator	2-3x faster than system malloc on small allocations
ArcSwap for TLS acceptor + metrics cache	Atomic pointer swap without mutex on hot path
Baseline-CPU build, SIMD chosen at run time	A binary runs on any processor of its architecture; ciphers, hashes and byte searches pick AES-NI / AVX2 / NEON code when the process starts (measured)
Pre-parsed upstream URIs	Scheme + authority parsed once at boot, only path set at runtime
DashMap for cache + rate limiter	Sharded concurrent map, avoids single-mutex bottleneck
FnvHashSet for CORS origins	O(1) origin lookup vs O(n) linear scan
Thread-local L1 with O(1) LRU	Index-based doubly-linked list, zero contention on cache hits
Same primitive for the route LRU	Per-worker thread-local cache (cap 256). Replaces a previous "first 256 then no more inserts" map that could be locked out by a flood of distinct paths.
Generation counter for L1/L2 coherence	Stale L1 entries detected via atomic compare, no broadcast needed
Singleflight via tokio::sync::watch	Receiver::wait_for inspects the current value at first poll, so a waiter that subscribes after the fetcher has already published completion still observes it — race that the previous Notify-based implementation could lose, leaving a waiter hung until the client timed out
SHA-256 leaf fingerprint for mTLS forward header	Replaces a 64-bit XOR-fold (which produced massive collision classes) emitted as X-Client-Cert-DN. Header renamed to X-Client-Cert-Fingerprint and prefixed sha256:; ACL via fingerprint is now safe.
Metrics are a process-global static, on purpose	metrics::METRICS is written directly from the cache, the proxy, the dispatcher and the workers, through sharded counters and fixed atomics: no AppState handle to thread through every call, no Arc clone, no indirection on the hot path. The cost is that a component cannot have counters of its own and that parallel tests share them: a test asserts that a counter rose, or reads it from a separate process (the integration tests do), never an exact value another test could move. One process serves one set of listeners, so a per-listener or per-instance scope has nothing to separate today; if zion is ever embedded more than once in a process, this is the thing to revisit (#532).
Differential histogram buckets	3 atomic ops per observation instead of 17 (prefix sums at render time)
OnceLock for WAF scanners + WS TLS	One Aho-Corasick automaton per WAF mode (balanced + aggressive); root cert store for upstream WS TLS — both built once on first hit, reused for process lifetime
Stack buffers for request ID + traceparent	Zero heap allocation on per-request headers
Arc<AutoBuilder> for HTTP builder	Per-connection clone is ref-count bump, not deep copy
io_uring single-shot accept (Linux)	Dedicated accept thread; one SQE re-submitted per connection
SO_BUSY_POLL (Linux)	Spin-poll NIC queue 50us for lower p99 latency
Semaphore for connection limit	[server] max_connections, or derived from memory (the cgroup limit when there is one): (MB / 4) * 1024 / 256 (~256 KB/conn), clamped 1k-100k

Concurrency model

Zion uses Tokio's multi-threaded runtime. Worker count is set to available CPU cores (N-1 on machines with >4 cores), pinned to physical cores via core_affinity. Each accepted connection is a spawned task. Total concurrent connections are bounded by a Tokio semaphore.

Optional features

Feature	Flag	Description
ACME auto-renewal	--features acme	HTTP-01 challenge, auto-renew via Let's Encrypt
JWT/OIDC auth	--features auth	Per-route JWT validation (HMAC, RSA, ECDSA, JWKS)
HTTP/3 QUIC	--features http3	UDP listener on same port, Alt-Svc advertisement
io_uring accept	--features io-uring-accept	Linux 5.19+, single-shot accept on a dedicated thread
NUMA-aware sharding	--features numa-aware	Per-NUMA-node DashMap shards (Linux multi-socket; single-shard fallback elsewhere)
Init wizard / dev mode	--features init	zion init / zion auto scaffolding + self-signed cert
Live TUI	--features tui	zion top dashboard
OpenTelemetry export	--features otel	OTLP trace exporter; not active yet, zion opens no spans (#640)
FIPS 140-3	--features fips	aws-lc-rs FIPS-validated backend
Sovereign geo	--features geo-ita / geo-eu	Baked-in IT/EU ASN+CIDR classification
AIMP mesh (experimental)	--features sovereign-aimp	Ed25519-signed fleet gossip of WAF/reputation
kTLS offload (experimental)	--features ktls	In-kernel TLS after handshake; wired, not yet exercised e2e in CI
ML-augmented WAF (experimental)	--features ml-waf	ONNX scorer on the WAF hot path; ships no bundled model

The release bundle is --features dist (= acme + init + auth). The full flag list and lean/cargo install defaults live in the README "Build flavors" section.

Build with multiple features:

cargo build --release --features "acme,auth,http3"