What 5,535 DolphinBench Runs Taught Us About Local AI Memory

1. Why DolphinBench Breaks Toy Haystacks & Our 5-Gate Clean-Room

Toy needle-in-a-haystack benchmarks hide a self-contained fact inside filler text and query the exact same nouns. Real workspace agents fail on stateful action execution under evolving rules. Released by Mem0 (mem0ai/dolphinbench), DolphinBench tests 600 live mock_mcp workspace tasks across 13,539 chronological sessions (1,500,265 o200k_base tokens) and three personas—Morgan (200 Founder/Ops tasks), Alex (200 Staff SWE tasks with 18 KB incident logs and deprecated flags), and Riley (200 Growth PM tasks)—plus 15 pure-deterministic counterfactuals. Passing requires retrieving standing policies that share zero nouns with today's terse prompt, resolving [Date: YYYY-MM-DD] temporal supersession, chaining list_* state lookups before mutating records, and withholding prohibited side effects.

Every run in our 9 × 615 = 5,535-trace suite is enforced by verify_zero_leakage.py across five gates:

1. Zero-LLM 0o444 SQLite Immutability: Ingests all 13,539 sessions in 153.4s with 0 LLM calls ($0.00), truncates WAL, applies chmod 0o444 + file:...?mode=ro URIs, and verifies pre/post SHA-256 digests across every 600-task run.
2. Static AST Clean-Room Scan: Verifies zero imports from grading rubrics, zero persona proper nouns, and zero test-ID branching.
3. Official 5-Method BenchmarkHarness Replay: Passes upstream python -m harness.export_submission --release upstream --check offline with 0 errors.
4. Mechanical Check Suite (599 Tasks / 1,310 Assertions): Audits deterministic tool arguments and mock_mcp state mutations without an LLM judge.
5. Counterfactual Causality Gate (15 Tasks, memory_enabled=True vs. False): Confirms causal lift from retrieved memory (+53.3 pp on both 30B and 120B).

2. Axis A: Zero-LLM Hybrid Memory vs. Raw Cosine (+12.50 pp SWE)

Cloud memory layers run paid LLM extraction over every message during ingestion, whereas naive vector RAG embeds whole turns and fetches top-k cosine neighbors. Holding Stock gpt-oss:120b, 1024d snowflake-arctic-embed2, the 4,200-token budget, and judges fixed, swapping single-query cosine (VanillaSnowflakeHarness) for SovereignMemoryStore lifts overall pass rate from 48.50% (291/600) → 55.83% (335/600, +7.33 pp).

• Morgan Ties (65.0% Vanilla vs. 64.0% Sovereign): Short, self-contained executive notes (flight seating, wire caps) already rank cleanly under dense cosine similarity.
• Alex (32.0% → 44.5%, +12.50 pp) & Riley (48.5% → 59.0%, +10.50 pp) Jump: SWE and PM logs mix 14 KB pasted incident packets with terse channel rules. SovereignMemoryStore fixes three failure modes deterministically: (1) 256/32 Raw-RAG windows (arXiv:2610.30755) prevent long turns from diluting pooled vectors or starving token budgets; (2) Porter-FTS5 BM25 + 4-Way RRF (k = 60) anchors exact symbols like dedup-v2 and #release-ops; and (3) directive/transition regex priors + chronological turn_index packing keep later-dated policy overrides ordered last, backed by active search_persona_memory calls.

# sovereign_memory.py — Read-only 0o444 URI guard + 4-Way Hybrid RRF + Raw-RAG window projection
def retrieve_persona_briefing(path: Path, cache: FrozenCache, query: str, q_vec: list[float], rrf_k: int = 60):
    if (path.stat().st_mode & 0o222) != 0:
        raise PermissionError(f"Refusing benchmark retrieval on writable SQLite store: {path}")

    fts_expr = build_fts5_or_query(query)  # Quoted Porter-stemmed token OR-clause
    conn = sqlite3.connect(f"file:{path}?mode=ro", uri=True, timeout=10.0)
    try:
        top_turn_bm25 = [int(r[0]) for r in conn.execute(
            "SELECT rowid - 1 FROM turns_fts WHERE turns_fts MATCH ? ORDER BY bm25(turns_fts) ASC LIMIT 128",
            (fts_expr,),
        ).fetchall()]
        top_win_bm25 = [int(r[0]) for r in conn.execute(
            "SELECT rowid - 1 FROM windows_fts WHERE windows_fts MATCH ? ORDER BY bm25(windows_fts) ASC LIMIT 128",
            (fts_expr,),
        ).fetchall()]
    finally:
        conn.close()

    # 4-Way RRF: Turn-Dense (1.0) + Turn-BM25 (1.15) + Win-Dense (0.85) + Win-BM25 (0.95)
    turn_rrf: dict[int, float] = {}
    for rank, (t_idx, _) in enumerate(top_turn_dense):
        turn_rrf[t_idx] = turn_rrf.get(t_idx, 0.0) + 1.00 / (rrf_k + rank + 1)
    for rank, t_idx in enumerate(top_turn_bm25):
        turn_rrf[t_idx] = turn_rrf.get(t_idx, 0.0) + 1.15 / (rrf_k + rank + 1)
    for rank, (w_idx, _) in enumerate(top_win_dense):
        for t_idx in cache.windows[w_idx]["turn_indices"]:
            turn_rrf[t_idx] = turn_rrf.get(t_idx, 0.0) + 0.85 / (rrf_k + rank + 1)
    for rank, w_idx in enumerate(top_win_bm25):
        for t_idx in cache.windows[w_idx]["turn_indices"]:
            turn_rrf[t_idx] = turn_rrf.get(t_idx, 0.0) + 0.95 / (rrf_k + rank + 1)
    # Cross-encoder rerank + salience boost -> knapsack-pack 4,200 tokens in chronological order.

3. Axis B: Parameter Scale Phase Transition (8B → 30B MoE → 120B MoE)

With retrieval frozen, every model evaluated on SovereignMemoryStore receives the byte-identical 4,200-token turn-1 briefing across all 600 tasks, isolating parameter scale across our 6-run Phase-1 matrix (3,690 traces across Node-02 Dual RTX 2080 Ti 22 GB NVLink, Node-01 Dual RTX 3060 24 GB, and 96 GB Blackwell G4 burst):

Stock hermes3:latest (8B Q4_0) did not fail on JSON tool syntax—it emitted valid tool calls on 593/600 tasks (66.44% mechanical task gate), yet collapsed to 4.50% (0/15 pure deterministic) when binding multiple policy constraints simultaneously. Stepping up to local Qwen3-Coder-30B-A3B (only 3.3B active parameters on Dual RTX 2080 Ti at $0.00) crosses that reasoning threshold immediately, hitting 80.33% under hermes3 and a suite-high 88.81% (532/599) mechanical compliance.

4. Axis C: Wave-3 Refusal Postmortem → Wave-4 Unified QLoRA + DPO

Our most useful discovery came from Run 3's failure, where applying our Wave-3 SRE Code-Auditor adapter to gpt-oss:120b dropped generalist accuracy from 55.83% → 36.50%, alongside a hidden Ollama template bug on 8B:

• Wave-3 Auditor Role Bias (30.0% Side-Effect Withholding): Hardcoding a zero-trust SRE Auditor SYSTEM prompt into the fused Modelfile plus bug-hunting SFT caused 15.6x over-calling of read-only search_persona_memory (701 vs. 45 calls) and 180/600 (30.0%) zero-mutation turns where the model asked for human approval instead of executing routine calendar/email tasks.
• Ollama's Silent Hermes-3 Go Template Bug ({{- else if .System }}): Inspecting ollama show --modelfile hermes3:latest explained why Stock 8B scored near zero—attaching MCP tools silently discarded the entire runtime SYSTEM prompt while duplicating <tool_call> branches on assistant turns:

{{- if .Tools }}
<|im_start|>system
You are a function calling AI model... <tools>{{ .Tools }}</tools><|im_end>
{{- else if .System }}   <-- BUG: .System is silently dropped whenever .Tools is non-empty!
<|im_start|>system
{{ .System }}<|im_end>
{{- end }}

5. Official openai/gpt-5.6-sol Leaderboard & 6/6 Queued Archives

We ran all three Wave-4 adapters (1,845 additional traces → 5,535 total verified Wave-3/Wave-4 ablation traces), graded all five open-weights Sovereign configurations across 600/600 tasks—plus a companion frontier scaling control pairing the byte-identical frozen 0o444 SovereignMemoryStore with Gemini 4 Argon (reasoning_effort="medium")—with the official openai/gpt-5.6-sol (reasoning_effort="medium") judge, verified offline via harness.export_submission --check, and queued all 6/6 archives on dolphinbench.ai:

• Wave-4 Beat Stock at Every Scale: Local 30B climbed 42.00% → 42.67% (252 → 256/600, 1,123/1,310 = 85.73% mechanical checks, +11 tasks on Riley) at $0.00 and 15.72s—outperforming MiniMax M3 + Hindsight (41.00% @ $158.33) and Claude Sonnet 5 + Honcho/Mem0 (35.83% / 32.33% @ $1,566–$1,831). 120B eliminated Wave-3 prose refusals (39.17% → 39.50%, 235 → 237/600), and 8B Q4_0 jumped 15.7x (7/600 → 110/600 = 18.33%, beating MiniMax M3 + Supermemory on Morgan 52 vs. 45).
• Why Local 30B (42.67%) Beats 120B (39.50%): Qwen3-Coder-30B-A3B chains multi-step list_* state discovery before writes on 320+ tasks (only 2 zero-tool turns), whereas gpt-oss:120b's Harmony alignment pauses to ask clarifying questions on 28–42 terse SWE prompts instead of inspecting workspace state first.

6. Honest Frontier Gaps, Argon Validation & Engineering Tradeoffs

1. Local -28.00 pp Open-Weights Gap vs. +18.50 pp Frontier Lead on Identical Memory: On open-weights MoEs (3B–5B active parameters), local $0.00 30B Wave-4 (42.67%, 256/600) trails cloud #1 Hermes + GPT 5.6 Luna + Mem0 (70.67% @ $96.21) by -28.00 pp, and scaling total experts alone in stock Qwen3-Coder-Next-80B (3.0B-active / 79.7B-total MoE) scores 38.50% (231/600). Conversely, pairing the exact same byte-identical 0o444 frozen SovereignMemoryStore with Gemini 4 Argon (reasoning_effort="medium") reaches 89.17% (535/600, +18.50 pp above #1 at $24.57 total cost)—proving the 46.50 pp gap was active multi-hop reasoning rather than retrieval, as detailed in our companion Paper #27 (Return on Memory) and validating @krusch/toolkit/cascade's L3 frontier escalation tier.
2. Judge Calibration Spread (hermes3 80.33% vs. gemma4:e4b 53.00% vs. gpt-5.6-sol 42.67%): Non-thinking local judges credit tool-family matches while gpt-5.6-sol penalizes any missing implicit field, yet rank order held across all judges (Wave-4 > Stock and 30B > 120B > 8B)—validating local Referees + 1,310 mechanical checks for $0.00 inner-loop iteration.
3. 8B Q4_0 Multi-Hop Ceiling (18.33%): Even after a 15.7x lift, 14.5% on Alex/Riley confirms 4-bit 8B models belong in L1 fast triage and single-hop dispatch, not multi-app state mutation.
4. Role-Neutral Weights Beat Persona Modelfiles: Moving persona identity to runtime prompts while training weights on capability primitives lets one fused GGUF serve both strict code audits and proactive assistant tasks without refusal drift.

7. Reproducibility, Compute Ledger (49.13 / 58.54 CUs) & Public Artifacts

# 1. Offline unit suites & 5-gate static AST / 0o444 clean-room verifier
.venv/bin/python -m unittest discover -s . -p "test_*.py" -v && .venv/bin/python verify_zero_leakage.py --static-only

# 2. Local 600-task + 15-counterfactual run on Dual RTX 2080 Ti ($0.00 / 0 CUs)
.venv/bin/python run_homelab_models_600.py --model qwen3-coder-30b-moe:latest --workers 4

# 3. Official openai/gpt-5.6-sol grading + offline upstream archive check
.venv/bin/python package_official_submissions.py --only wave4_30b --workers 12
PYTHONPATH=upstream .venv/bin/python -m harness.export_submission --release upstream --check submissions/sovereign_qwen3_coder_30b_wave4_qlora.zip

• Compute & Cost Ledger: 49.13 Colab Pro CUs across the 9-run Wave-3/Wave-4 ablation & training suite (+ 9.41 = 58.54 CUs total including the Qwen3-Coder-Next-80B scale check; 5,513.20 CUs remaining, 0 active sessions, 0.00/hr); $24.57 API spend for the 600-task Gemini 4 Argon frontier scaling run (Paper #27); and 0.00 CUs / $0.00 API spend across 153.4s ingestion, local 30B Stock & Wave-4 (42.67%) on Node-02 Dual RTX 2080 Ti, and local Referees on Node-01 Dual RTX 3060 + Node-03 RTX 2080 Ti.
• Public Datasets & Open Weights: krush420/sovereign-g4-benchmarks (dolphinbench_600 split) · krush420/sovereign-router-arena (Tab 5) · krush420/sovereign-code-audit-sft (18,131 rows) · Wave-4 adapters (30B, 120B, 8B) · github.com/kruschdev/krusch-stack.