How an Open-Weights Sovereign Stack Stacks Up Across 317,000+ Benchmark Queries

1. Condensed 6-Pillar External Scorecard (10 Suites)

Rather than inventing custom micro-benchmarks or grading models with themselves, we ran each subsystem of krusch-stack against ten established academic and industry benchmarks inside an isolated 96 GB NVIDIA RTX PRO 6000 Blackwell (sm_120) clean-room VM and local 1024d snowflake-arctic-embed2 loopback (zero outbound cloud API calls on every open-weights run).

2. How the Stack Compares Against Three Architecture Archetypes

A. vs. Standard Open-Source "Wrapper" Stacks

Typical open-source agent stacks glue LangChain or LlamaIndex splitters (fixed 500-token windows) to a single vector store, stuff every MCP tool schema into the system prompt, pass raw untrusted tool outputs straight back into the model, and allow coding or SQL agents to execute unverified 1-shot generations. Our controlled ablations isolate why each of those defaults breaks at scale:

• Out-of-Distribution Routing Collapse & Preflight Latency: Supervised routers fit in-distribution benchmarks well, then degrade sharply when domains shift. On ACL 2026 LLMRouterBench, RouteLLM (8.75%) and MLP (9.25%) collapse on unseen ARC-AGI reasoning tasks while zero-shot structural complexity gating + isotropic 8K snowflake-arctic-embed2 centroids reach 25.25% and #1 overall Pareto distance (0.4269). Independently on Martian RouterBench (arXiv:2403.12031 Sec. 5.2, 36,497 0-shot + 36,483 5-shot rows), KNN (0.6648), SVM (0.6363), and MLP (0.5572) all collapse below non-predictive ZeroRouter (0.6941) under Leave-One-Suite-Out OOD, whereas KruschZeroShotRouter (0.7268 0-shot / 0.7455 5-shot) and KruschCascadeRouter (0.7083 / 0.7389) rank #1 and #2 above ZeroRouter while also taking #1 In-Distribution Micro AIQ (0.7488 / 0.7695). On multi-step Aider Polyglot (n=60), replacing a 569.51ms 7B LLM preflight classifier with deterministic compiler/test failure routing (0.395ms, 1,442x faster) lifts Pass@2 from 21.67% to 51.67% (2.38x).
• Structure-Preserving Indexing vs. Fixed Token Windows: Holding snowflake-arctic-embed2 (1024d) constant on LegalBench-RAG, replacing fixed 160-token windows (11.70% F1) with KruschNexus hierarchical Section → Clause parsing + contextual section-path headers + local mxbai-rerank-large-v2 cross-encoder reranking lifts span-exact Macro F1 by +128.5% (26.74%). Across all 10/10 ACL 2025 CoIR tasks (2,389,123 docs & 162,216 queries), krusch-git AST-spine + identifier subtoken hybrid RRF reaches 56.40% Macro NDCG@10 (+7.42 pp over BM25 48.98%, lifting codefeedback-mt +8.00 pp to 60.23% and synthetic-text2sql +4.22 pp to 48.71%) while beating pure dense vectors across all 8/8 RepoBench-R splits.
• 2PC State Grounding, Deferred MCP Discovery & Taint-Tracked Tool Guards: Staging code patches behind TruncationGuard + Verify-on-Stop heals 72 failed Aider Polyglot suites (+32.00 pp), while read-only node:sqlite FK/cell-value introspection + 2PC SQL self-healing lifts BIRD-SQL Mini-Dev from 26.00% to 40.20% EX (100.00% valid SQL across 500/500 queries) at just 0.09 extra calls/query. Mounting a 64-tool Enterprise MCP hub on all 2,251 UC Berkeley BFCL v3 Live queries, createDeferredToolBridge + schema-repair guard cuts prompt schema tokens by -80.67% (7,894 → 1,526 tok/turn) while lifting Micro accuracy from 59.88% to 66.28% (+144 queries) and irrelevance abstention from 67.23% to 84.69% (+17.46 pp). And on ETH SpyLab AgentDojo v1.2.2 (1,046 episodes), taint-tracked untrusted boundary delimiting + deterministic sink-provenance policy floors drive indirect prompt-injection ASR from 19.28% (183/949) down to 0.00% (0/949) with zero benign utility degradation.

B. vs. Commercial Cloud Router, Memory, Code & Legal Stacks

• Cloud Routing Economics: In official RouteWorks/RouterArena GitHub CI, krusch-cascade-router achieves 0.7883 (80.72% accuracy) at $0.2117 / 1K queries—surpassing commercial routers such as Azure-Model-Router (0.7042), Martian (0.5756 @ $4.83/1K, 22.8x costlier), and NotDiamond (0.5729 @ $4.10/1K, 19.4x costlier). On Martian RouterBench (0-shot ID, n=10,953 eval), KruschCascadeRouter retains 98% of GPT-4 accuracy at $9.89 vs. $36.15 (-72.63% cost, 504.7 µs/q—2.83x faster and half the cost of KNN $18.71) and 95% of GPT-4 at $6.08 (-83.18%). On live multi-GPU generation, speculative SSE logprob/loop stream abort (Path B) cuts 120B TTFT by -48.0% (1,694.5ms → 880.7ms) and compute cost by -37.4% even with all 325 aborted streams (58,743 wasted tokens) strictly billed.
• Contradiction & 10M Multi-Session Memory: Packing a fixed ~8K-token slice (0.66% of a 10M-token chat) on zero-Docker node:sqlite, krusch-context-mcp v2.0.0 shows zero E2E degradation from 100K → 500K → 1M (70.60% → 71.82% → 72.86% on hermes3) and beats cloud past.dev (openai/gpt-5.6-luna) on Contradiction Resolution across all 4/4 splits (92.50% Stock / 98.13% SRE-QLoRA vs. 78.13% at 100K; 89.64% vs. 78.57% at 500K; 77.68% vs. 75.36% at 1M; 78.75% vs. 73.13% at 10M) and on 10M Multi-Session Reasoning (29.54% Qwen / 38.33% Hermes vs. 28.17%).
• Air-Gapped Code & Legal Retrieval Precision: On MAUD merger agreements (1,676 queries), KruschNexus hits 33.95% character-exact Recall@2 locally—exceeding published OpenAI text-embedding-3-large + Cohere Rerank Recall@64 (28.28%) while retrieving 32x fewer spans. On ACL 2025 CoIR (10/10 tasks, 2.39M docs), krusch-git's local 1024d hybrid RRF (56.40% NDCG@10) tops proprietary Voyage-Code-002 (56.26%) and OpenAI-ada-002 (53.66%; see §3 context-window disclosure).

C. Clean-Room Auditability vs. Unverified Leaderboard Claims

Every run in our benchmark archive passes 68 / 68 automated integrity gates: (1) loopback air-gap verification (127.0.0.1:11434, cloud API keys unset), (2) runtime JavaScript Proxy traps, opaque RPC session tokens, and Python pre-eval column stripping on ground-truth answer/SQL/qrel/injection fields that block oracle access before inference, (3) RestrictedUnpickler + SafeZipExtractor guards, read-only DatabaseSync SQL enforcement, full-corpus 2,389,123-doc anti-shrinkage checks, and zero hardcoded attacker literals on AgentDojo, (4) mandatory Exercism test un-skipping + pristine test SHA-256 checks, and (5) zero LLM self-judging (deterministic compiler, SQL set-equivalence, AST bijection, and math verifiers on 9 benchmarks; dual cross-family open-weights referees on BEAM).

3. Where Frontier Cloud Stacks Still Beat Us (Honest Gaps & Open PRs)

• 1. Aggregate BEAM E2E Reader Ceiling (70.6%–73.5% Open-Weights vs. 86.2% Hindsight / 92.1% past.dev): Three distinct factors drive this gap:
  • Cross-family referees vs. cloud self-judging: We grade open-weights gpt-oss:120b (5.1B active MoE) with two independent cross-family open-weights judges (qwen2.5:32b and hermes3:latest, 81%–88% binary agreement). By contrast, past.dev's 92.1% uses openai/gpt-5.6-luna as both reader and judge.
  • Exact-position ordering rubrics: On Event Ordering (100K), krusch-context-mcp v2.0.0 retrieves 81.19% of gold turns, yet qwen2.5:32b scores 9.68% while hermes3:latest scores 43.17% (vs. 100% past.dev) because qwen2.5:32b awards 0.0 whenever a 10-event timeline shifts by one slot.
  • Passive supersedes_id = NULL ingestion on Knowledge Update: Despite 97.50% retrieval recall at 100K, E2E QA lands at 37.5%–52.5% (vs. 86%–92.5% past.dev) because our zero-heuristic benchmark loader ingests raw chat turns with supersedes_id = NULL, leaving outdated historical turns active in the 8K window. In production IDE sessions, coding agents explicitly invoke krusch_context_revise({ action: "supersede" }) to mark superseded facts inactive at the SQLite/Postgres layer.
• 2. Multi-Turn Frontier Reasoning Upper Bound on Coding & Text-to-SQL (51.56% Polyglot & 40.20% BIRD-SQL vs. 65%–88%+ Cloud): Lifting open-weights gpt-oss:120b at $0 API cost from 19.56% → 51.56% (116/225) on Aider Polyglot matches Claude 3.5 Sonnet (51.6%) and beats GPT-4o (23.1%), and lifting BIRD-SQL Mini-Dev from 26.00% → 40.20% EX (100% valid SQL at 1.09 calls/query) recovers +14.20 pp without fine-tuning—yet 109/225 hard algorithmic C++/Rust/Go/Java suites and complex multi-join SQL aggregates still separate single-retry open-weights MoEs from multi-turn/multi-candidate frontier reasoning pipelines (o3, gpt-5, Claude 4, Gemini 4 Argon at 75%–88%+ coding and 65%–75%+ multi-candidate SQL). That gap is why our Cascade Router retains an L3 frontier escalation tier for the hard tail.
• 3. Monolingual English Feature Variance on Held-Out CJK OOD (0.0811 leave-chinese-out 0-Shot): On Martian RouterBench 0-Shot Leave-One-Suite-Out OOD, learned KruschCascadeRouter ranks #1 among learned routers on 7-suite Macro AIQ (0.6873) and #2 overall Micro AIQ (0.7083, behind zero-shot KruschZeroShotRouter at 0.7268), yet drops to AIQ = 0.0811 on held-out chinese (n=236, pulling Macro-9 to 0.6156) because training exclusively on English 0-shot suites produces std(cjk_ratio) ≈ 0 against the 1e-4 z-score variance floor. In-distribution 0-shot chinese ranks #1 (0.4880) and 5-shot OOD chinese recovers to 0.3059; zero post-hoc test tuning was applied.
• 4. ACL 2025 CoIR Context-Length Protocol Difference (8,192-Token Production Window vs. Table 3 512-Token Truncation): In the ACL 2025 CoIR paper's Table 3, all 10 baseline retrievers (including Voyage-Code-002 56.26% and E5-Mistral 7B 55.18%) were evaluated under a uniform 512-token truncation cap so legacy 512-token encoders could be compared side by side. Our krusch-git run (56.40% 10-task Macro NDCG@10 across all 2,389,123 unshrunk docs) evaluates snowflake-arctic-embed2:latest at its native 8,192-token production context window + AST spine headers—reflecting real repository indexing while accounting for part of the margin on long-document tasks such as apps and stackoverflow-qa.
• 5. Open Upstream Leaderboard Submissions (Explicit Status):
  • RouteWorks/RouterArena (PR #169): Scored 0.7883 by the official RouteWorks github-actions[bot] CI workflow across all 8,400 queries + 420 robustness perturbations; PR #169 is open and awaiting maintainer merge onto the upstream leaderboard.
  • withmartian/routerbench (kruschdev/routerbench:feat/krusch-cascade-router): Drop-in AbstractRouter implementation (KruschCascadeRouter + KruschZeroShotRouter) and 4-regime ID/OOD harness published on our public fork branch and ready for upstream PR submission.
  • ETH SpyLab AgentDojo, UC Berkeley BFCL & Aider Polyglot: Our full 1,046-episode AgentDojo v1.2.2 (0.00% ASR), 2,251-query BFCL v3 Live, and 225/225 Aider Polyglot runs execute the official upstream suites inside clean-room harnesses; upstream Python adapter PR playbooks for ethz-spylab/agentdojo and Aider-AI/polyglot-benchmark remain unsubmitted next steps.
• 6. Upstream Dataset Hygiene Notes: Two upstream rows in RepoBench-R ([5522, 7591] in python_cff/test_easy) carry out-of-bounds gold_snippet_index pointers and are deterministically quarantined (47,998 / 48,000 evaluated). In Martian RouterBench routerbench_5shot.pkl, 28 upstream rows have missing model cells across the 11-model matrix and are deterministically filtered so all 6 routers evaluate the identical 36,483 / 36,511 complete rows. In UC Berkeley BFCL v3 BFCL_v3_live_relevance.json, upstream ships a duplicate row ID (live_relevance_3-3-0 at lines 3 and 4 with distinct prompts), deterministically disambiguated as live_relevance_3-3-0__dup2 so all 18 / 18 rows (2,251 / 2,251 Live queries) execute without overwrite. In ICLR 2026 BEAM 100K, upstream Conversation 14 Knowledge Update Q1 asks about a nonexistent "running shoe" topic; our verifier asserts score == 0.00 (canary_conv14_ku_q1_zero: true).

4. Raw Artifacts & 60-Second Cryptographic Verification

Every raw per-query JSON audit log (21 files, SHA-256 pinned) and 13 Hugging Face Dataset Viewer splits (35 files / 350.90 MiB) are published under CC-BY-4.0 at krush420/sovereign-g4-benchmarks, alongside the interactive Pareto & trajectory explorer at krush420/sovereign-router-arena and repository-level reproduction docs in kruschdev/krusch-stack.

# Clone the raw clean-room audit logs & verify all SHA-256 digests in <60s
git clone https://huggingface.co/datasets/krush420/sovereign-g4-benchmarks
cd sovereign-g4-benchmarks

python3 -c '
import hashlib, json, pathlib
rows = [json.loads(l) for l in pathlib.Path("data/scorecard_summary.jsonl").read_text().splitlines()]
for r in rows:
    if r["audit_log_file"].startswith("raw_audit_logs/"):
        digest = hashlib.sha256(pathlib.Path(r["audit_log_file"]).read_bytes()).hexdigest()
        assert digest == r["audit_log_sha256"], f"Mismatch: {r[\"audit_log_file\"]}"
        print("VERIFIED:", r["audit_log_file"], digest[:16])
'