Return on Memory: Why Test-Time Reasoning Unlocks Zero-Distillation Agentic Retrieval on Official DolphinBench (89.17%, #1 Overall)

1. The Write-Time Distillation Fallacy & Formal Definition of Return on Memory (RoM)

Prevailing production memory layers—including Mem0, Honcho, Hindsight, and Supermemory—rely on write-time LLM distillation: every incoming conversation session is fed through an extraction model that emits condensed atomic facts, entity-relation triples, or persona summaries, while discarding or demoting verbatim turn transcripts. On Official DolphinBench (13,539 chronological sessions totaling 1,500,265 o200k_base tokens across three workspace personas), write-time LLM distillation imposes an immediate, non-recoverable tax before a single user task is executed: $61.67 for Mem0 (64.1% of its $96.21 run budget), $109.76 for Honcho (76.8% of $142.85), and $313.86 for Supermemory (90.8% of $345.64).

Worse than the upfront dollar cost is the information-theoretic defect: write-time distillation forces a lossy compression step without conditioning on the future test-time query. When an executive says in March, "Keep investor updates under four bullets unless Q3 ARR slips below plan," and an engineering incident log in August casually mentions Q3 ARR tracking 4% under target, a query-agnostic write-time extractor frequently flattens the March note into a generic style preference ("Prefers 4-bullet investor updates") or drops the conditional trigger entirely. Once compressed out of the index at write time, no amount of test-time reasoning can recover the missing premise.

In What 5,535 DolphinBench Runs Taught Us About Local AI Memory and our ICLR 2026 BEAM 10M benchmark, we took the opposite architectural stance—Zero-Distillation Append-Only Verbatim Raw-RAG inside SovereignMemoryStore:

• Verbatim Dual-Granularity Indexing (0 LLM Calls, $0.00, 153.4s Total): Every session is appended verbatim with its explicit [YYYY-MM-DD] narrative timestamp and segmented into overlapping 256-token Raw-RAG windows (32-token stride, 7,935 windows across all three personas). Both full turns and 256/32 windows are embedded locally in a single 153.4s pass (11.33 ms/turn) using 1024d snowflake-arctic-embed2:latest alongside dual SQLite Porter-stemmed FTS5 BM25 tables (turns_fts, windows_fts).
• 4-Way Reciprocal Rank Fusion + Deterministic Salience & Reranking: At read time, SovereignMemoryStore fuses Turn-Dense (weight 1.00), Turn-BM25 (1.15), Window-Dense (0.85), and Window-BM25 (0.95) via RRF(k = 60), applies regex directive and state-transition priors (catching temporal overrides like "switched from", "no longer", "effective immediately"), cross-encoder reranks candidates, and packs a 4,200-token turn-1 briefing in strict chronological turn_index order—augmented by an active search_persona_memory MCP tool for iterative multi-hop query reformulation.
• Cryptographic 0o444 Read-Only Freezing: After ingestion, WAL journals are truncated and morgan.sqlite (44.7 MB), alex.sqlite (53.2 MB), and riley.sqlite (53.6 MB) are locked chmod 0o444 and opened exclusively via file:...?mode=ro URIs, guaranteeing byte-for-byte identical memory state across every model we evaluate.

Formalizing Return on Memory (RoM)

When open-weights agents (8B through 120B MoE) plateaued between 18.33% and 42.67% on Official gpt-5.6-sol grading against these frozen stores, standard RAG intuition would blame the zero-LLM index for lacking LLM-extracted entity graphs. To test whether the ceiling resided in the memory substrate or the reader's test-time reasoning policy, we formalize Return on Memory. Let M_0 denote a fixed, read-only memory index (0o444 SQLite) and let pi_R denote an agentic tool-use policy parameterized by active test-time reasoning compute budget R:

Because M_0, sovereign_memory.py, sovereign_harness.py, system instructions, tool schemas, and the initial 4,200-token retrieved turn-1 briefing per task are held 100% invariant across all seven 600-task evaluations, any non-zero Delta_RoM(R_2, R_1 | M_0) is causally attributable purely to test-time reasoning and active query reformulation—establishing an empirical lower bound on the information already preserved inside M_0.

2. Law I — The Return-on-Memory Scaling Curve & The Active-Parameter MoE Bottleneck

Evaluating seven distinct model configurations across all 600 Official DolphinBench tasks (4,200 verified execution traces, every one graded by the upstream openai/gpt-5.6-sol judge at reasoning_effort="medium") uncovers a striking three-regime scaling law on identical 0o444 SovereignMemoryStore checkpoints:

Full 7-Run Controlled Sovereign Matrix vs. Official DolphinBench Cloud Leaderboard

Table 1 reports all seven clean-room SovereignMemoryStore evaluations alongside the top six Official DolphinBench cloud leaderboard stacks under identical openai/gpt-5.6-sol (reasoning_effort="medium") grading:

Why 80B-A3B (38.50%) and 120B-A5B (39.50%) Plateau Alongside 30B-A3B (42.67%)

At first glance, seeing Qwen3-Coder-Next-80B (79.7B total parameters, 38.50%) and gpt-oss:120b (116.8B total parameters, 39.50%) fail to surpass Qwen3-Coder-30B-A3B (30.5B total parameters, 42.67%) looks paradoxical. Mechanistically, it is a direct consequence of sparse Mixture-of-Experts routing over verbatim retrieved context:

• Total Experts Store Dormant World Knowledge; Active Parameters Bound Cross-Context Composition: Scaling routed FFNs from 30.5B → 79.7B → 116.8B expands parametric trivia (rare library syntax, encyclopedic facts), which DolphinBench deliberately neutralizes because every persona preference and project state lives in retrieved episodic context, not pretraining weights. Per-token working memory—simultaneously attending across five chronologically separated Raw-RAG windows, resolving date overrides, and binding four constraint fields into a single JSON tool call—is governed by active parameters per token (3.0B to 5.1B) and test-time reasoning depth.
• The 92.33% Empirical Retrieval-Sufficiency Proof: Across the union of our six open-weights runs (Oracle@6), at least one open model solves 416/600 (69.33%) tasks. Adding Gemini 4 Argon solves 138 additional tasks that went 0/6 across every open-weights run, pushing the 7-run empirical union (Oracle@7) to 554/600 (92.33%)—with only 19 tasks solved by an open model where Argon missed a minor formatting or judge nuance. Thus, SovereignMemoryStore's zero-LLM index was never capped at 42%; +46.50 pp of retrieved signal sat waiting in the prompt window for sufficient test-time reasoning to compose it.

3. Law II — Superlinear Return on Memory Across Multi-Fact Compositional Depth (1.81x → 3.88x)

If test-time reasoning unlocks retrieved raw evidence primarily by composing dispersed constraints rather than memorizing syntax, then the Marginal Reasoning Dividend Delta_RoM should grow monotonically with the number of Load-Bearing Facts (LBF) required per task. Annotating all 600 upstream test specifications by exact ground-truth load-bearing fact count—LBF = 1 (n = 264), LBF = 2 (n = 165), LBF = 3 (n = 96), and LBF = 4+ (n = 75, spanning 4 to 9 simultaneous facts)—confirms a steep, superlinear divergence:

Cross-Session Evidence Span (HistSess) & Active Query Reformulation

Stratifying the same 600 tasks by Historical Session Span (HistSess)—the number of distinct episodic sessions across which a task's ground-truth evidence is dispersed—mirrors the LBF scaling law:

• Single-Session Evidence (HistSess = 1): Stock Qwen3-Coder-30B-A3B achieves 48.7%, whereas Gemini 4 Argon reaches 88.2% (+39.5 pp)—capturing subtle single-session constraints like negative prohibitions and exact recipient formatting.
• Dual/Triple-Session Synthesis (HistSess = 2–3): Stock 30B slides to 41.6% as competing dates appear in the 4,200-token briefing, while Gemini 4 Argon climbs to 92.2% (+50.6 pp).
• Wide Cross-Session Synthesis (HistSess = 4+): Stock 30B drops by more than half to 21.2%, whereas Gemini 4 Argon holds 82.5% (+61.3 pp, 3.89x).
• Iterative search_persona_memory Reformulation (1.75 Calls/Task on Argon): Beyond synthesizing the initial 4,200-token briefing, Gemini 4 Argon actively issues 1,050 targeted follow-up search_persona_memory calls (1.75/task)—probing secondary entities discovered inside list_* app state (e.g., looking up a teammate's dietary restriction or PTO window only after discovering who is on an invite list)—compared to 1.20/task on 30B Wave-4 (722 calls), 1.16/task on 80B Stock (695), 0.46/task on 120B Wave-4 (273), and just 0.06/task on Stock gpt-oss:120b (36 calls).

4. Law III — Flattening the Corpus-Entropy Cliff (Morgan → Alex → Riley)

DolphinBench's three workspace personas form a natural corpus-entropy and fact-density gradient:

1. Morgan (Founder / Executive Ops): 3,400 sessions · 336 ground-truth facts · 1.77 mean LBF/task — crisp, low-entropy executive directives and travel/calendar rules.
2. Alex (Staff Software Engineer): 5,011 sessions · 299 ground-truth facts · 1.97 mean LBF/task — high structural noise (+47.4% more sessions, multi-kilobyte stack traces, deprecated feature flags, and incident postmortems).
3. Riley (Growth Product Manager): 5,128 sessions · 423 ground-truth facts · 2.50 mean LBF/task — maximum relational entropy and multi-stakeholder temporal supersession across experiment rollouts, pricing tiers, and cross-channel norms.

Plotting accuracy across Morgan → Alex → Riley exposes a structural vulnerability shared by every write-time LLM distillation architecture on the Official Leaderboard:

Why Write-Time LLM Distillation Decays by -15.5 to -20.0 pp Under High Entropy

Notice that GPT 5.6 Luna is already a frontier reasoning model—yet pairing it with Mem0, Hindsight, Honcho, or Supermemory still triggers a -15.50 to -20.00 pp cliff on Alex and Riley, whereas our zero-LLM Raw-RAG index paired with Gemini 4 Argon widens its lead over Mem0 from +11.50 pp on Morgan to +20.50 pp on Alex and +23.50 pp on Riley. Three structural mechanisms explain this gap:

• Premature Schema Commitment Destroys Conditional Caveats: Write-time extractors summarize turns into fixed subject-predicate-object or bullet schemas before knowing which edge condition will matter months later. On Riley, stakeholder exceptions ("cc Legal on enterprise tier discounts >15%, except renewals pre-approved in #deal-desk") routinely lose their exception clauses during write-time extraction.
• Temporal Supersession Erasure: Merging facts into mutable profile nodes or de-contextualized bullets strips exact [narrative_date] progression cues and conversational hedging. Verbatim 256/32 Raw-RAG preserves every dated turn intact and orders retrieved chunks strictly by chronological turn_index, enabling test-time reasoning to trace policy evolution end-to-end.
• Implicit Interpersonal & Channel Boundaries: On Alex, technical norms differ between #incident-war-room, PR review comments, and direct messages with external vendors. Verbatim windows preserve speaker tone, channel headers, and code identifiers (caught by Porter-FTS5 BM25) that abstractive summarization blurs together.

5. Law IV — Cost–Accuracy Pareto Dominance & Cascade Routing Implications

Combining zero-LLM local ingestion ($0.00) with high-hit-rate prefix caching during test-time reasoning establishes a two-point global Pareto frontier across the entire Official DolphinBench cost–accuracy plane—anchoring both $0.00 sovereign local execution (42.67%) and $24.57 global #1 accuracy (89.17%):

Exact Token & Prefix-Cache Accounting Ledger ($24.565 Total)

Every token across all 2,293 Gemini 4 Argon model responses in sovereign_gemini_4_argon.zip is priced directly by upstream harness.costing:

6. Clean-Room Thought-Signature Bridge (argon_openai_bridge.py), Submission Ledger & Reproducibility

A strict methodological requirement of our 7-run study was keeping sovereign_memory.py (SHA-256: 16700bc047b0a0e8...) and sovereign_harness.py (SHA-256: 4531e3efa4e110e3...) 100% byte-identical across every local and frontier run—zero Argon-specific prompt edits, zero retrieval weight changes, and zero harness forks.

However, running multi-turn OpenAI-formatted MCP tool loops against Gemini 4 Argon's OpenAI-compatible endpoint introduces a subtle protocol requirement: every assistant tool call emitted by a thinking Gemini model carries a cryptographic extra_content.google.thought_signature token that must be echoed back on subsequent conversation turns so latent reasoning state remains continuous across tool steps, whereas generic OpenAI client loops strip unknown fields or prefix step IDs (s{step}_{idx}_). We solved this transparently at the transport layer via a lightweight, zero-dependency localhost proxy (argon_openai_bridge.py on 127.0.0.1:11439):

# argon_openai_bridge.py (127.0.0.1:11439) — Zero-mutation protocol bridge preserving sovereign_harness.py SHA-256
STEP_PREFIX_RE = re.compile(r"^s\d+_\d+_")

def normalize_outbound_body(body: dict[str, Any]) -> dict[str, Any]:
    out = dict(body)
    sys_parts, rest_msgs = [], []

    for m in out.get("messages") or []:
        role = m.get("role")
        # 1. Coalesce leading system/developer blocks (SYSTEM_PROMPT + 4,200-tok Briefing) -> 56.5% prefix cache hit
        if role in ("system", "developer") and not rest_msgs:
            if isinstance(m.get("content"), str) and m["content"].strip():
                sys_parts.append(m["content"])
        else:
            m_copy = dict(m)
            # 2. Re-attach cached google.thought_signature across multi-turn MCP tool steps
            if role == "assistant" and m_copy.get("tool_calls"):
                patched_tcs = []
                for tc in m_copy["tool_calls"]:
                    tc_copy = dict(tc)
                    if "extra_content" not in tc_copy:
                        cid = str(tc_copy.get("id") or "")
                        bare_id = STEP_PREFIX_RE.sub("", cid)
                        fn = tc_copy.get("function") or {}
                        fkey = canonical_fn_key(str(fn.get("name") or ""), fn.get("arguments"))
                        with _SIG_LOCK:
                            extra = _EXTRA_BY_ID.get(cid) or _EXTRA_BY_ID.get(bare_id) or _EXTRA_BY_SIG.get(fkey)
                        if extra:
                            tc_copy["extra_content"] = extra
                    patched_tcs.append(tc_copy)
                m_copy["tool_calls"] = patched_tcs
            rest_msgs.append(m_copy)

    out["messages"] = ([{"role": "system", "content": "\n\n".join(sys_parts)}] if sys_parts else []) + rest_msgs
    # 3. Strip JSON Schema draft keys ($schema, additionalProperties, $defs) unsupported by OpenAPI validator
    if out.get("tools"):
        for t in out["tools"]:
            if isinstance(t.get("function"), dict) and "parameters" in t["function"]:
                t["function"]["parameters"] = sanitize_gemini_schema(t["function"]["parameters"])
    return out

Offline Verification & Reproducibility Commands

# 1. Verify 5-gate static AST clean-room & 0o444 SQLite SHA-256 immutability
.venv/bin/python -m unittest discover -s . -p "test_*.py" -v && .venv/bin/python verify_zero_leakage.py --static-only

# 2. Launch clean-room thought-signature bridge & execute 600-task Gemini 4 Argon evaluation
.venv/bin/python argon_openai_bridge.py --host 127.0.0.1 --port 11439 &
OPENAI_BASE_URL=http://127.0.0.1:11439/v1 .venv/bin/python run_full_600.py --model gemini-4-argon --reasoning-effort medium

# 3. Replay & audit official gpt-5.6-sol archive (535/600 = 89.17%, $24.57 total) with 0 errors
PYTHONPATH=upstream .venv/bin/python -m harness.export_submission \
  --release upstream --check submissions/sovereign_gemini_4_argon.zip