NOVA V2

macOS image studio: Build the native app with bash scripts/macos/build.sh, then open build/Nova.app. Nova Studio mirrors the MOSS-Studio workspace and runs the Nova image pipeline as its bundled server, with discovery of existing local credentials. See the macOS app guide.

 _   _  _____  _   _   ___   _   _  _____ 
| \ | ||  _  || | | | / _ \ | | | |/ __  \
|  \| || | | || | | |/ /_\ \| | | |`' / /'
| . ` || | | || | | ||  _  || | | |  / /  
| |\  |\ \_/ /\ \_/ /| | | |\ \_/ /./ /___
\_| \_/ \___/  \___/ \_| |_/ \___/ \_____/

The Compute Ranch: Autonomous Standalone Rust Inference Pipeline & Shard Routing Substrate

Enterprise-grade AI inference coordination, multi-account SWRR credential pooling, lock-free continuous token buckets, resumable broadcast ring buffers, and resilient duplex WebSocket streaming in pure Rust.

---

⚡ Quickstart • 🏛️ Architecture • 💡 Philosophy • 📄 Whitepaper • 🤝 Handoff • 🔌 Gateway APIs • 👥 Shard Fleet • 📊 Performance Audit • 📦 Client SDKs • ⚙️ Config Matrix • 🐳 Deployment • 🧪 56 Tests • 📚 Wiki & Docs • 🛡️ Maintainers • 🔍 Post-Mortems • 🚀 Setup • 📊 Metrics • 🔒 Security • 📜 License

---

⚡ Quick Navigation

Getting Started & Architecture	APIs & Integrations	Core Mechanics & Fleet	Ops, Governance & Wiki
• Why NOVA V2?	• Gateway API Reference	• Smooth Weighted Round Robin	• Knowledge Base Wiki
• Feature Comparison Matrix	• Chat Completions Spec	• Continuous Token Buckets	• Architecture Decision Records (ADRs)
• 60-Second Quickstart	• Resumable SSE Streams	• Canary Circuit Breaker	• Blameless Post-Mortems
• Multi-Language Clients	• Native Rust Client SDK	• Duplex WebSocket Relay	• Maintainers Charter
• Substrate Architecture	• Python OpenAI SDK	• Resumable Broadcast Ring Buffer	• Anti-Scope-Creep Policy
• Engineering Philosophy	• Technical Whitepaper	• Stakeholder Handoff	• Maintainers Charter
• Project Structure	• LangChain Integration	• Atomic SHA-256 Idempotency	• Setup & Tuning Directory
• Local 60s Setup	• Shard Fleet Enrollment	• Atomic Shard Leasing	• Prometheus Metrics Catalog
• 56-Test Verification Suite	• cURL & Wire Protocol	• Forensic Audit Report	• Documentation Linter Gate

---

Why NOVA V2? (The Compute Ranch Mandate)

NOVA is an inference company. Its sole core responsibility is to provide deterministic, ultra-low-latency, resilient inference for designated models.

V1 accumulated architectural obligations that obscured this purpose: UI dashboards, database stored procedures, Supabase migrations, and productization concepts.

❌ The Accidental Complexity of V1:
┌─────────────────────────┐  ┌─────────────────────────┐  ┌─────────────────────────┐
│   Node.js Event Lag     │  │   Supabase / DB Locks   │  │ Accidental UI Dashboards│
│ V8 GC pauses spike TTFT │  │ 150+ SQL migrations &   │  │ Frontend templates and  │
│ from 17ms to 36ms+.     │  │ stored procedure hops.  │  │ CSS bundled in gateway. │
└─────────────────────────┘  └─────────────────────────┘  └─────────────────────────┘
┌─────────────────────────┐  ┌─────────────────────────┐  ┌─────────────────────────┐
│  High Baseline Memory   │  │ In-Process Embedding    │  │ Kubernetes Pilgrimage   │
│ 102 MB idle RSS memory; │  │ Temptation to embed     │  │ Complex orchestration   │
│ 140 MB under load.      │  │ gateway into apps.      │  │ for simple daemon.      │
└─────────────────────────┘  └─────────────────────────┘  └─────────────────────────┘

⚡ The Standalone Rust Substrate (V2):
┌─────────────────────────┐  ┌─────────────────────────┐  ┌─────────────────────────┐
│   Tokio Work-Stealing   │  │ Zero External Database  │  │ Pure Wire Protocol API  │
│ 12,408 req/sec; 1.36ms  │  │ 100% in-memory native   │  │ Standard HTTP/1.1 REST, │
│ TTFT; microsecond p99.  │  │ state; zero SQL bloat.  │  │ SSE, and WebSockets.    │
└─────────────────────────┘  └─────────────────────────┘  └─────────────────────────┘
┌─────────────────────────┐  ┌─────────────────────────┐  ┌─────────────────────────┐
│  Featherweight Footprint│  │ Anti-Embedding Invariant│  │ Single Static Binary    │
│ 9.2 MB idle RSS memory; │  │ Isolated out-of-process │  │ Zero runtime deps;      │
│ 19.6 MB under load.     │  │ daemon; clean boundary. │  │ instant zero-config run.│
└─────────────────────────┘  └─────────────────────────┘  └─────────────────────────┘

NOVA V2 ("The Compute Ranch") preserves every working capability of V1 while rebuilding the core pipeline in pure, idiomatic Rust:

• 🌊 Algorithmic Credential Pooling: Smooth Weighted Round Robin (SWRR) balances traffic smoothly across heterogeneous accounts.
• ⚡ Continuous Token-Bucket Governance: Real-time token accrual and sliding-window limits per tenant with sub-millisecond overhead.
• 🛡️ Canary Circuit Breakers: Dissects 429 quota exhaustion vs. 3-strike 5xx faults, quarantines failing shards, and admits isolated canary probes.
• 🔄 Duplex WebSocket Transport: Bridges real-time OpenAI Codex responses over WebSocket with a 3-layer timeout protection hierarchy.
• 🔁 Resumable Event Stream Bus: Bounded ring buffer per job with monotonic sequence numbers, allowing reconnecting SSE clients to resume via Last-Event-ID.
• 🔒 Atomic SHA-256 Idempotency: Collapses concurrent identical prompts into the in-flight stream (isAttachedDuplicate: true), achieving up to 16,000 req/sec fanout.
• 🛡️ Bounded TTL Memory Pruning: Periodic background Tokio worker evicts aged completed/failed jobs and hashes, preventing memory leaks under 24/7 load.
• 🏛️ Autonomous Standalone Daemon: Strict anti-embedding invariant. It runs as an independent operating system daemon communicating exclusively over wire protocols.

---

Feature Comparison Matrix

Capability	Generic Proxy (Envoy / Traefik)	Generic LLM Gateway (LiteLLM / Portkey)	V1: TypeScript / Node.js	V2: Rust (The Compute Ranch)
Language & Runtime	C++ / Go	Python	TypeScript (Node.js 20+)	Pure Rust (Tokio 1.40 + Axum 0.7)
Sustained Throughput	High (HTTP only)	~200 – 600 req/s	460 – 553 req/s	4,320 – 12,408 req/s (9.4× – 22.8× faster)
Mean Time to First Token (TTFT)	N/A	~25 – 60 ms	17.3 – 36.5 ms	1.36 – 1.96 ms (10× – 26× lower)
Baseline Resident Memory (RSS)	~30 – 60 MB	~120 – 250 MB	101.3 MB	9.2 MB (11× lower footprint)
OpenAI-Compatible Streaming Wire	❌ (Raw proxy)	✅ (HTTP only)	✅ (SSE + WebSocket)	✅ (SSE + Duplex WS + Resumption)
Smooth Weighted Round Robin (SWRR)	⚠️ (Basic round robin)	⚠️ (Static weights)	✅ (TypeScript SWRR)	✅ (Pure Rust Atomic SWRR)
Two-Phase Continuous Token Buckets	❌ (Basic rate limit)	⚠️ (External Redis)	✅ (In-Memory Engine)	✅ (Continuous Leaky Bucket)
Circuit Breakers with Canary Slots	⚠️ (Passive TCP)	⚠️ (Threshold trip)	✅ (Canary state machine)	✅ (Canary Isolation + 429 Backoff)
Duplex WebSocket Upstream Relay	❌ (Blind TCP)	❌ (HTTP-to-HTTP)	✅ (Node ws package)	✅ (Tokio-Tungstenite Async Stream)
Resumable SSE Stream Buffer	❌ (None)	❌ (None)	⚠️ (Partial)	✅ (Last-Event-ID Token Replay)
Atomic In-Flight Stream Attachment	❌ (None)	⚠️ (Cache response)	✅ (Deduplication table)	✅ (Atomic SHA-256 Broadcast Bus)
External DBMS Dependency	❌ (N/A)	⚠️ (Redis / Postgres)	❌ (Postgres required)	✅ ZERO External DBMS (100% Standalone)
In-Process Embedding Invariant	N/A	⚠️ (Embedded library)	⚠️ (Accidental coupling)	✅ STRICT Anti-Embedding Mandate

---

Quickstart

1. Build from Source

# Clone and enter directory
cd nova-v2

# Build optimized release binary
cargo build --release

2. Run the Daemon

# Start on port 3001 with loopback open (synthetic development engine)
./target/release/nova-v2

# Or launch with custom port, API authentication, and upstream WebSocket endpoint
./target/release/nova-v2 \
  --port 3001 \
  --api-key "secret-nova-key" \
  --admin-key "admin-secret-key" \
  --ws-endpoint "wss://chatgpt.com/backend-api/codex/responses" \
  --shard-concurrency 64

3. Send Your First Inference Request

curl -X POST http://127.0.0.1:3001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer secret-nova-key" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Explain Rust ownership in two sentences."}],
    "stream": false
  }'

---

Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                       CONSUMING CLIENT APPLICATIONS                         │
│       (Web Frontends, Microservices, Python SDKs, AI Agents, CLI)           │
└──────────────────────────────────────┬──────────────────────────────────────┘
                                       │ Standard HTTP/1.1 & SSE
                                       │ (Port 3001)
                                       ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                    NOVA V2 STANDALONE RUST DAEMON                           │
│                                                                             │
│  [ Ingress & Authentication ]                                               │
│    • Axum HTTP Router with zero routing regex overhead                      │
│    • Bearer API Key validation (constant-time token check)                  │
│    • Continuous Token-Bucket Rate Governor (per tenant)                     │
│    • SHA-256 Idempotency Store (duplicate stream attachment)                │
│                                                                             │
│  [ Resource Pool & Shard Balancer ]                                         │
│    • Smooth Weighted Round Robin (SWRR) Balancer                            │
│    • Circuit Breaker (429 cooldowns, 3-strike 5xx, canary slots)            │
│    • Shard Fleet Substrate (atomic write-lock lease allocation)             │
│                                                                             │
│  [ Relay Engine & Buffer ]                                                  │
│    • Duplex WebSocket Upstream Transport (Codex protocol)                  │
│    • 3-Layer Timeout Protection (Handshake, TTFT, Intra-chunk)              │
│    • Ring-Buffer Resumable Event Stream Bus (Last-Event-ID)                 │
│    • Bounded TTL Memory Pruning Worker (evicts aged completed jobs)         │
│    • Prometheus Metrics & Health Probes                                     │
└──────────────────────────────────────┬──────────────────────────────────────┘
                                       │ Upstream Duplex WebSocket
                                       ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                     UPSTREAM MODEL INFERENCE SHARDS                         │
│                    (OpenAI, Codex, Heterogeneous Fleet)                     │
└─────────────────────────────────────────────────────────────────────────────┘

For complete architectural deep dives, see:

• Rust Standalone Substrate Architecture
• Smooth Weighted Round Robin Formulation
• Continuous Token Bucket Rate Governance
• Circuit Breaker, 429 Quarantine & Canary Slots
• Duplex WebSocket Upstream Relay Transport
• Resumable SSE Broadcast Ring Buffer Bus
• Atomic SHA-256 Idempotency & Stream Attachment
• Atomic Shard Lease Check & Allocation
• Sovereign Shard Store & Autonomous Credential Discovery

---

📂 Project Structure

nova-v2/
├── Cargo.lock
├── Cargo.toml
├── CHANGELOG.md                    # Keep a Changelog v2.0.0 release notes
├── CODE_OF_CONDUCT.md              # Contributor Covenant v2.1
├── CONTRIBUTING.md                  # Development setup & PR checklist
├── LICENSE                          # MIT License
├── MAINTAINERS.md                   # Project governance & maintainers charter
├── README.md                        # Master documentation & benchmark specification
├── SECURITY.md                      # Security disclosure policy & defense guarantees
├── docs/
│   ├── README.md                   # Knowledge Base & Engineering Wiki Master Index
│   ├── adrs/                       # 10 Architecture Decision Records (ADR-0001 to ADR-0010)
│   ├── api/                        # Complete REST/SSE wire protocol specifications
│   ├── architecture/               # 10 Deep dive specifications on core mechanics
│   ├── benchmarks/                 # Forensic audit, methodology & reproduction guide
│   ├── handoff/                    # Formal executive & technical stakeholder handoff
│   ├── maintainers/                # 8 Governance policies & operational standards
│   ├── observability/              # Prometheus metric catalog & distributed tracing
│   ├── philosophy/                 # The Compute Ranch Mandate & 7 engineering pillars
│   ├── playbooks/                  # 3 Triage playbooks for live incidents
│   ├── post-mortems/               # 3 Blameless incident & audit retrospectives
│   ├── runbooks/                   # 3 Step-by-step operational runbooks
│   ├── sdks/                       # Client integration guides (Rust, Python, LangChain, cURL)
│   ├── setup/                      # Setup guides (Local 60s, Systemd, Tokio tuning)
│   └── whitepaper/                 # Formal systems architecture technical whitepaper
├── scripts/
│   └── lint-docs.sh                # Automation shell wrapper for documentation gate
├── src/
│   ├── bin/
│   │   ├── benchmark_audit.rs      # Multi-scenario audit benchmark binary
│   │   ├── lint_docs.rs            # Native zero-dependency documentation & link linter
│   │   ├── nova-v2.rs              # Production daemon binary entrypoint
│   │   ├── nova_ctl.rs             # Interactive onboarding & fleet administration CLI wizard
│   │   └── run_benchmarks.rs       # In-memory performance harness
│   ├── pool/                       # SWRR balancer, circuit breakers, rate governor, persistent shard store
│   ├── relay/                      # Axum HTTP routes, WebSocket duplex upstream relay, embedded web cockpit
│   ├── balance.rs                  # SWRR balancer, TokenBucket rate governor & circuit types
│   ├── circuit.rs                  # Canary circuit breaker with 429 & 5xx fault isolation
│   ├── config.rs                   # CLI flag parsing & runtime environment configuration
│   ├── credentials.rs              # Autonomous candidate discovery & JWT claim validator
│   ├── gateway.rs                  # Axum HTTP routes, Bearer auth, SSE streaming handlers
│   ├── lib.rs                      # Public library exports & integration facade
│   ├── main.rs                     # Standalone CLI binary entrypoint
│   ├── pipeline.rs                 # In-flight job registry, SSE bus, TTL pruner & WebSocket relay
│   └── types.rs                    # OpenAI DTOs, shard descriptors, and protocol types
└── tests/
    ├── integration_tests.rs        # End-to-end Axum HTTP/SSE test suite (8 tests)
    ├── relay_tests.rs              # Dual-pool, persistent store, and WebSocket relay tests (34 tests)
    └── unit_tests.rs               # Balancer, Circuit, Token Bucket, Pipeline tests (14 tests)

---

Gateway APIs

1. Chat Completions (POST /v1/chat/completions)

Full OpenAI wire protocol compatibility for streaming and non-streaming inference.

Decoupled Asynchronous Streaming (stream: true):

Returns HTTP 202 Accepted with stream URLs:

{
  "id": "job_1792602969058_d0232c0d",
  "object": "chat.completion.job",
  "status": "queued",
  "model": "gpt-4o",
  "streamUrl": "/v1/jobs/job_1792602969058_d0232c0d/stream",
  "statusUrl": "/v1/jobs/job_1792602969058_d0232c0d",
  "isAttachedDuplicate": false,
  "pollAfterMs": 2000,
  "createdAt": 1792602969
}

Synchronous Non-Streaming (stream: false):

Awaits full completion and returns standard chat.completion payload with usage token statistics.

2. Resumable SSE Stream (GET /v1/jobs/:id/stream)

Connects to the live token emission bus. Supports Last-Event-ID: <seq> to replay dropped chunks from the in-memory ring buffer.

3. Job Cancellation (POST /v1/jobs/:id/cancel)

Instantly aborts in-flight inference, closes the upstream WebSocket connection, and broadcasts event: cancelled.

4. Shard Fleet Management & Frictionless Onboarding

Dynamically enrolls, inspects, and re-weights inference accounts into the SWRR balancer without daemon restarts.

• Interactive Terminal Wizard:

  nova-ctl onboard

• Autonomous Credential Inspection:

  nova-ctl auth inspect

• Zero-Downtime Shard Enrollment API (POST /api/shards/enroll): Enrolls or hot-swaps tokens and weights in-memory with write-ahead sync to ~/.nova/shards.json.
• One-Shot Autonomous Onboarding API (POST /api/shards/onboard): Discovers host credentials, evaluates plan tier, models, and TTL, and enrolls the optimal shard in a single call.
• Hot Token Refresh (POST /api/shards/{id}/refresh): Refreshes OAuth access tokens in-flight without interrupting ongoing inference streams.

5. Health & Observability (/health, /health/ready, /metrics)

Standard health checks and Prometheus-compatible metrics exposition format.

See the complete Gateway API Reference.

---

Performance Audit & Benchmark Evidence

In accordance with the Executive Commission from MAJO (Executive Chair) and EGG (Principal), the benchmark harness and implementation were subjected to independent verification.

Apples-to-Apples Comparison: 100 Unique Requests (Concurrency 10)

Both engines executed identical workloads: 100 distinct requests with unique Idempotency-Key headers, establishing 100 distinct upstream WebSocket streams:

Metric	V1: TypeScript / Node.js	V2: Rust (The Compute Ranch)	Verified Delta
Sustained Throughput	460.77 req/sec	4,320.88 req/sec	$9.4\times$ higher throughput ⚡
Total Test Duration	217 ms	23 ms	$9.4\times$ faster duration
Mean TTFT	20.95 ms	1.96 ms	$10.7\times$ lower latency ⚡
P50 TTFT	17.35 ms	1.71 ms	$10.1\times$ lower latency
P99 TTFT	64.16 ms	4.23 ms	$15.2\times$ lower latency
Mean Request Latency	20.98 ms	2.29 ms	$9.2\times$ lower latency
P99 Request Latency	64.20 ms	4.64 ms	$13.8\times$ lower latency
Initial Memory RSS	101.31 MB	9.23 MB	$11.0\times$ lower footprint
Peak Memory RSS	139.25 MB	19.61 MB	$7.1\times$ lower peak memory
RSS Delta Under Load	+37.94 MB	+10.38 MB	$3.7\times$ lower memory churn

High-Throughput In-Memory Stream Attachment (Duplicate Fanout)

When identical concurrent prompts omit unique keys, V2's atomic SHA-256 idempotency engine attaches listeners to the in-flight broadcast ring buffer:

• Throughput: 8,300.85 req/sec (reproduces the originally reported $8,383\text{ req/s}$ figure)
• Duration: 12 ms | Mean TTFT: 1.07 ms | Mean Latency: 1.18 ms

Sustained High-Load Test: 1,000 Requests (Concurrency 20)

• V1 (Node.js): 544.25 req/sec (single-threaded event-loop saturates, V8 GC pauses)
• V2 (Rust): 12,408.30 req/sec ($22.8\times$ higher sustained throughput)

Read the full Executive Audit Report and Methodology Analysis.

---

Client SDKs

Python (Official OpenAI SDK)

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:3001/v1", api_key="secret-nova-key")

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Explain quantum computing in one sentence."}]
)
print(response.choices[0].message.content)

Rust (Native Asynchronous Client)

use nova_v2::client::NovaClient;
use nova_v2::relay::handlers::{ChatCompletionRequest, ChatMessage};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = NovaClient::new("http://127.0.0.1:3001", Some("secret-nova-key".into()), None);

    let res = client.chat(ChatCompletionRequest {
        model: Some("gpt-4o".into()),
        messages: vec![ChatMessage {
            role: "user".into(),
            content: "Hello from native Rust!".into(),
        }],
        stream: false,
        max_tokens: Some(100),
        temperature: Some(0.7),
    }).await?;

    println!("Response: {}", res.choices[0].message.content);
    Ok(())
}

See the Client SDK Catalog.

---

Configuration Matrix

NOVA V2 is configured via CLI arguments or environment variables:

Argument	Environment Variable	Default	Description
--host	NOVA_HOST	127.0.0.1	Network interface to bind HTTP/SSE server.
--port	NOVA_PORT	3001	TCP port to listen for incoming requests.
--api-key	NOVA_API_KEY	None (open)	Optional Bearer token required for /v1/* requests.
--admin-key	NOVA_ADMIN_KEY	None (open)	Optional Bearer token required for /api/shards/*.
--ws-endpoint	NOVA_WS_ENDPOINT	None (synthetic)	Upstream OpenAI responses WebSocket endpoint.
--shard-concurrency	NOVA_SHARD_CONCURRENCY	32	Default max in-flight requests per enrolled shard.
--tenant-rate-limit	NOVA_TENANT_RATE_LIMIT	2000.0	Token bucket refill rate (tokens/sec) per tenant.
--tenant-burst-capacity	NOVA_TENANT_BURST_CAPACITY	2000	Token bucket burst capacity ceiling per tenant.
--buffer-retention-ms	NOVA_BUFFER_RETENTION_MS	300000	Duration (ms) completed jobs remain in memory (5 min).
--ws-connect-timeout-ms	NOVA_WS_CONNECT_TIMEOUT_MS	5000	Layer 0 WebSocket handshake timeout in milliseconds.
--ws-ttft-timeout-ms	NOVA_WS_TTFT_TIMEOUT_MS	15000	Layer 1 Time to First Token timeout in milliseconds.
--ws-intra-chunk-timeout-ms	NOVA_WS_INTRA_CHUNK_TIMEOUT_MS	10000	Layer 3 intra-chunk streaming timeout in milliseconds.
--state-path	NOVA_STATE_PATH	None	Optional path to persist fleet state snapshot JSON.

---

Production Deployment & Operational Ergonomics

NOVA V2 is designed as a cloud-native, self-contained inference substrate deployable as a bare-metal Systemd daemon, a hardened container, or an enterprise Kubernetes cluster.

1. Embedded Web Management Console (/dashboard)

Every NOVA V2 daemon features an embedded, zero-dependency, real-time dark-mode operational console served directly at http://localhost:3001/dashboard (and /):

• Universal Command Palette (⌘K / Ctrl+K): Keyboard-first navigation across shards, rollouts, telemetry, and docs.
• 4-Pillar Ergonomics: Fleet Topology, Canary Rollout Controller, Live Prometheus Telemetry, and Interactive Inference Playground with real-time SSE token streaming and TTFT stopwatch.
• Zero Runtime Dependencies: Compiles directly into the static binary via include_str!, eliminating external asset servers or database dependencies.

2. Multi-Stage Hardened Docker Container (Dockerfile)

# Build multi-stage hardened container (<30MB runtime, unprivileged UID 10001)
docker build -t nova-v2:2.0.0 .

# Run with container healthcheck and isolated port
docker run -d --name nova-v2 \
  -p 3001:3001 \
  -e NOVA_HOST=0.0.0.0 \
  -e NOVA_PORT=3001 \
  nova-v2:2.0.0

3. Turnkey Production Stack (docker-compose.yml)

Spawns a full production/staging cluster including primary relay, isolated canary relay, NGINX edge reverse proxy with unbuffered SSE pass-through, and Prometheus metrics scraping:

docker compose up -d

4. Kubernetes Cloud-Native GitOps Suite (k8s/)

• Zero-Downtime Draining: Injects a 15-second preStop hook to allow in-flight SSE streams to finish before pod termination.
• Automated Canary Analysis (ACA): Routes $10%$ traffic to isolated canary deployments with automated rollback on circuit breaker trips.
• High-Availability Security Profile: readOnlyRootFilesystem: true, drop: ["ALL"], runAsNonRoot: true, PodDisruptionBudget minAvailable: 2, and HPA concurrency autoscaling.

kubectl apply -f k8s/configmap.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/ingress.yaml
kubectl apply -f k8s/hpa.yaml
kubectl apply -f k8s/pdb.yaml

5. Linux Systemd Service (/etc/systemd/system/nova.service)

[Unit]
Description=NOVA V2 Standalone Rust Inference Pipeline
After=network.target

[Service]
Type=simple
User=nova
Group=nova
ExecStart=/usr/local/bin/nova-v2 \
  --port 3001 \
  --shard-concurrency 64 \
  --buffer-retention-ms 300000
Restart=always
RestartSec=3s
LimitNOFILE=1048576

# Sandboxing
ProtectSystem=strict
ProtectHome=true
NoNewPrivileges=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Read the full Production Deployment Strategy Guide, ADR-0008: Canary Deployments, and Production Runbook.

---

Testing & Verification

NOVA V2 maintains a comprehensive unit, integration, regression, and persistent store test suite:

# Run all 56 unit, integration, and relay regression tests
cargo test

# Run the native Rust documentation gate
cargo run --release --bin lint_docs

# Run the release benchmark in matched unique mode
cargo run --release --bin benchmark

# Run the release benchmark in deduplication fanout mode
BENCH_MODE=dedup cargo run --release --bin benchmark

# Run the full multi-scenario audit harness
cargo run --release --bin audit_harness

Verified Test Breakdown:

• Unit Tests (14 passed in 0.02s): SWRR smoothness, candidate filtering, continuous token buckets, sliding windows, circuit breaker 3-strike rules, 429 quota backoff, canary slots, token ring buffers, JobStore TTL eviction, atomic concurrent leasing.
• Integration Tests (8 passed in 0.16s): Auth enforcement, synchronous completion, streaming SSE completion, upstream WebSocket relay, idempotency attachment, job cancellation regression, upstream handshake failure recovery, memory retention pruning.
• Relay & Sovereign Shard Tests (34 passed in 0.28s): Dual-pool router, persistent shard store atomic writes, JWT claim extraction, candidate discovery quality ranking, runtime 401 revocation trapping, circuit breaker auto-quarantine, and hot-token refresh.
• Documentation Gate (73 Markdown docs, 290+ links tested, 0 broken, 0 warnings).

---

License

This project is licensed under the MIT License - see the LICENSE file for details.

BILLIONS MUST COMPUTE. EVIDENCE PRESERVED. THE COMPUTE RANCH IS OPERATIONAL. 🦀⚡