Benchmarking Agent Memory Without Cheating: Closed-System ICLR 2026 BEAM (100K–10M)

🛡️ 1. De-Rigging Agent Memory Benchmarks

Conversational memory benchmarks are easy to game unintentionally. Before running final evaluations, we audited our harness across three adversarial passes (19 fixes) and locked five anti-gaming checks inside benchmarks/verify_zero_leakage.js:

⚖️ 2. 105-Query, 4-Domain Embedding Bake-Off

Legacy bge-large:latest truncates inputs at 512 tokens (~1,150 chars). We benchmarked four open-weights Ollama embedders at 1,024d across 105 queries spanning Statutes, Contracts, Polyglot Code, and Agent Skills:

snowflake-arctic-embed2:latest gained +6.38 pp Macro Recall@1 for only +46 MB VRAM (zero autoregressive KV cache at 8K context), becoming the default across krusch-context-mcp and @krusch/toolkit/embeddings.

📐 3. Anisotropic BERT vs. Isotropic InfoNCE Calibration

Swapping in snowflake-arctic-embed2 immediately raised pure dense recall but initially cost -4 pp on hybrid Recall@10 because BERT and InfoNCE encoders occupy fundamentally different cosine ranges:

• Anisotropic BERT (bge-large): Noise sits at 0.48–0.55; true paraphrases sit at 0.68–0.85.
• Isotropic InfoNCE (snowflake-arctic-embed2 + query: ): Noise centers at ~0.00 (-0.03 to +0.08); true paraphrases land at 0.35–0.55, below BERT-tuned hybrid gates.

export function calibrateQuerySimilarity(rawCosine, options = {}) {
    if (!Number.isFinite(rawCosine) || rawCosine <= 0) return 0;
    if (rawCosine >= 1) return 1;
    const { model, queryVec = null } = options || {};
    if (queryVec && _isSparseSyntheticVector(queryVec)) return rawCosine;

    const base = _normalizeModelName(model);
    if (base.startsWith('snowflake-arctic-embed2') || base.startsWith('qwen3-embedding')) {
        if (rawCosine <= 0.12) return rawCosine * 3.0;                       // [0.00, 0.12] -> [0.00, 0.36]
        if (rawCosine <= 0.38) return 0.36 + ((rawCosine - 0.12) / 0.26) * 0.32; // [0.12, 0.38] -> [0.36, 0.68]
        if (rawCosine <= 0.60) return 0.68 + ((rawCosine - 0.38) / 0.22) * 0.16; // [0.38, 0.60] -> [0.68, 0.84]
        return 0.84 + ((rawCosine - 0.60) / 0.40) * 0.16;                    // [0.60, 1.00] -> [0.84, 1.00]
    }
    return rawCosine;
}

🔒 4. Hermetic Single-Box Topology (96 GB Blackwell)

Every run executed inside a cold-room NVIDIA RTX PRO 6000 Blackwell (97,887 MiB VRAM) box with all four open-weights models resident in GPU memory simultaneously and zero external network or cloud API calls during inference:

┌───────────────────────────────────────────────────────────────────────────────────┐
│  HERMETIC SINGLE-BOX CLOSED SYSTEM · NVIDIA RTX PRO 6000 BLACKWELL (97.9 GB VRAM) │
│  OLLAMA_HOST=127.0.0.1:11434 · GEMINI_API_KEY="" · Zero .env · Cold-Room Fetch    │
├───────────────────────────────────────────────────────────────────────────────────┤
│  1. EMBEDDER  : snowflake-arctic-embed2:latest  (689 MB  · 8,192 ctx · 165,559x)  │
│  2. READER    : gpt-oss:120b                    (64.0 GB · 16,384 ctx · MXFP4)    │
│  3. JUDGE #1  : qwen2.5:32b (Alibaba Dense 32B) (21.0 GB · 4,096 ctx · temp=0)    │
│  4. JUDGE #2  : hermes3:latest (Nous Llama 8B)  ( 5.5 GB · 4,096 ctx · temp=0)    │
├───────────────────────────────────────────────────────────────────────────────────┤
│  Co-Located Footprint: ~91.2 / 97.9 GB (100% GPU · Zero CPU Offload · Zero Self-Grading) │
└───────────────────────────────────────────────────────────────────────────────────┘

📊 5. 100K Retrieval Recall & 3-Campaign Ablation

Table A compares primary-turn recall against upstream source_chat_ids across all 360 ground-truth 100K probes under an identical ~8,000-estimated-token budget (680/440/320-char condensation; ~67.9 Hybrid turns + epistemic headers vs. ~76.0 Pure Dense turns):

Separating Engine Upgrades from the 8K Embedder (n = 400/400)

Why Recall@10 (+5.26 pp cumulative) matters most: Live IDE Turn-1 hydration runs at limit_tokens = 4000 alongside invariants and blockers — fitting only 15–25 episodic memories rather than 68 benchmark turns.

🏆 6. 100K End-to-End QA, 120B-SRE QLoRA & Bottleneck Diagnosis

Across all 400/400 100K probes, qwen2.5:32b and hermes3:latest reached 88.00% (352/400) binary agreement (80.50% within ±0.25):

Published Mohammadta/BEAM (100K, n = 400) Leaderboard Context

Table C: In-Place 120B-SRE QLoRA Fusion Ablation (400/400)

Fusing our rank-16 BF16 SRE code-audit adapter (krush420/gpt-oss-120b-sre-lora) into gpt-oss:120b in 2.00s trades open-ended narrative verbosity (SUM -8.73 / -14.54 pp) for sharper state-mutation precision — pushing Contradiction Resolution to 98.13% (+20.00 pp above past.dev):

Why Stronger Embeddings Dipped 3 Categories on 100K (-2.93 pp)

• Semantic Neighbor Distractors on Abstention (73.75% → 61.25%): Unanswerable probes name real chat entities while asking about an unstated attribute; richer 8K neighborhoods tempt the Reader to hedge rather than refuse.
• Passive Ingestion on Knowledge Update (97.50% Recall → 37.50% QA): Blind benchmark rules forbid supersede links, packing both stale Turn #12 and updated Turn #148 side-by-side (whereas live IDE agents call krusch_context_revise(action: "supersede")).
• Exact-Count Event Ordering Rubrics (9.68% Qwen / 43.17% Hermes): Rubrics demand "ONLY three [or five] macro-events"; listing 4 granular milestones scores 0.0 on qwen2.5:32b. Excluding EO and passive KU, Stock 120B averages 74.43% / 76.92% across the other 320 probes.

📈 7. Multi-Split Scaling: 500K, 1M & 10M (All 2,000/2,000 Probes on SQLite)

Without changing a byte of v1.9.0 retrieval code, we scaled across 500K (700 Qs), 1M (700 Qs), and BEAM-10M (200 Qs, 10,435 turns/chat) while keeping the Reader budget fixed at ~8K tokens:

• Pure Dense Breaks Past 500K: Once an 8K window packs ≤ 12.7% of history, hybrid scoring + Porter FTS5 beats Pure Dense on every long split (+2.70 pp at 500K, +1.91 pp at 1M, +1.27 pp at 10M; +23.00 pp on 10M IE).
• Zero Context Rot up to 1M: Scaling conversation length 13× in tokens at fixed prompt cost produced zero E2E drop (70.60% → 71.82% → 72.86% on hermes3; Knowledge Update actually rose +27.5 pp from 100K to 10M as top-K competition pushed stale early turns out of the window).
• 4/4 Split Sweep on Contradiction Resolution: Deterministic conflict headers + Stock gpt-oss:120b beat cloud past.dev across all four splits (92.50 / 89.64 / 77.68 / 78.75% vs. 78.13 / 78.57 / 75.36 / 73.13%).
• SQLite Was Never the Bottleneck: Node 22 node:sqlite + 4 KB Float32Array blobs scanned 100% of 10,435-turn chats in < 300 MB RSS (152.3 cold embeds/sec).

🚀 8. Multi-Scale Retrieval Upgrade: krusch-context-mcp v2.0.0

Targeting the 10M leaf-turn bottlenecks diagnosed above, krusch-context-mcp v2.0.0 (commit 694ff34, 124/124 tests) adds three zero-LLM, zero-RPC stages inside the unchanged 5-verb contract and ~8K budget:

• Two-Level Episode Rollups (src/episode-rollup.js): Extractive 64-turn/session digests + salience-weighted Float32Array(1024) centroids (O(1) SQLite watermark cache; stripTurnCitations prevents leaf Turn-ID spoofing) emit macro timeline panoramas and pin standing USER: directives.
• Gated 2-Hop Residual Bridges (src/retrieval-bridge.js): Orthogonal Gram-Schmidt residual vectors (vbridge = normalize(vhop1 − 0.75⟨vhop1, vq⟩vq)) + disjoint-session FTS5 expansion rescue missing multi-session facets, rank-capped at 0.84 / 0.76 below Hop-1 Top-10 anchors.
• log10(N) Attribute Abstention (src/retrieval-epistemic.js): Checks ±1-turn USER: stem co-occurrence with corpus-scale cosine thresholds so 10,000-turn entity distractors (0.48–0.59 raw cosine) still trigger grounding warnings.

Table H: 4-Split Retrieval (v1.9.0 vs. Calibrated v2.0.0, All 2,000/2,000 Probes)

Table I: Closed-System BEAM-10M Head-to-Head (v1.9.0 vs. v2.0.0, n = 200/200, 81.00% Judge Agreement)

🧪 9. Cryptographic Provenance & One-Command Reproduction

Every dataset shard, retrieval module, model weight digest, and 2,000/2,000 audit log (v1.9.0 and v2.0.0) is SHA-256 verified offline:

# Clone krusch-context-mcp v2.0.0 and run all 124 SQLite unit + honesty tests (zero Docker)
git clone https://github.com/kruschdev/krusch-context-mcp.git
cd krusch-context-mcp && npm install && npm test

# Verify 5/5 zero-leakage checks, SHA-256 digests & 2,000/2,000 Dual-Judge tallies (v1.9.0 & v2.0.0)
npm run bench:beam:verify && npm run bench:beam:verify:long
node benchmarks/verify_zero_leakage.js --audit-log=benchmarks/g4_closed_system_v200/beam_100k_v200_audit_log.json
node benchmarks/verify_zero_leakage_long.js --split=500K --audit-log=benchmarks/g4_closed_system_v200/beam_500k_v200_audit_log.json
node benchmarks/verify_zero_leakage_long.js --split=1M --audit-log=benchmarks/g4_closed_system_v200/beam_1m_v200_audit_log.json
node benchmarks/verify_zero_leakage_long.js --split=10M --audit-log=benchmarks/g4_closed_system_v200/beam_10m_v200_audit_log.json

# Reproduce hermetic single-box runs on a 96GB Blackwell GPU
./scripts/run_beam_closed_g4.sh && ./scripts/run_beam_long_closed_g4.sh