The 15 Production Agentic Systems: Reference Architecture, State-Machine Invariants, and Benchmark Proofs Across a Sovereign Stack

1. Executive Matrix: 15 / 15 Production Agent Systems Mapped

Every row below corresponds to a non-negotiable failure boundary encountered when scaling autonomous coding, SRE, and legal/enterprise agents past single-user notebooks. Rather than scattering ad-hoc scripts across services, we unify these 15 subsystems across five composable repositories—with zero-dependency Node 22 (node:sqlite + node:crypto) parity inside krusch-stack and high-concurrency Python 3.12 asyncio / Pydantic v2 parity inside krusch-agentic-proxy.

2. Deep-Dive Teardown I — Event-Driven Concurrency, Typed Swarm State Machines & 2PC Persistence (#1, #3, #5, #6, #7, #13)

Why Naive while True Loops and Raw-String Handoffs Collapse

Tutorial agents fail under real load for three structural reasons: (1) synchronous HTTP or tool I/O inside an event loop pins the worker thread so a single stalled endpoint freezes 100 unrelated sessions; (2) passing free-form natural-language strings between a "Researcher" and a "Coder" invites progressive specification loss and lateral prompt injection; and (3) holding Human-in-the-Loop (HITL) approvals in process RAM loses pending state whenever a container deploys or restarts.

Bounded 100+ Task Concurrency & Per-Host Circuit Breakers (#1, #13)

In krusch-agentic-proxy (src/orchestrator.py), AsyncAgentOrchestrator bounds concurrent in-flight agent trajectories via asyncio.Semaphore(128), wraps every model/tool hop inside asyncio.wait_for deadlines with full-jitter exponential backoff, offloads synchronous OS/SQLite tools via await asyncio.to_thread(...), and isolates unhealthy model hosts behind a three-state (CLOSED → OPEN → HALF_OPEN) per-endpoint circuit breaker. Web acquisition (#13) follows the same latency discipline: Tier-0 pooled HTTP/REST probes execute first in <80 ms, escalating to headless Chromium (@krusch/toolkit/browser) only when status codes (403/429) or empty SPA root containers (<div id="root"></div>) prove client-side JavaScript rendering is required.

Strict Pydantic v2 Tool Routers & Frozen Multi-Agent Contracts (#3, #6)

Every tool invocation in StrictPydanticToolRouter validates arguments against a Pydantic v2 BaseModel configured with ConfigDict(extra="forbid", frozen=True). When a model emits an unknown key or wrong scalar type, the router catches ValidationError, serializes exact JSON-path error diagnostics back to the model, and converges within max_retries=3—healing 9 / 14 live schema faults on UC Berkeley BFCL v3. Across multi-agent workflows, TypedMultiAgentSwarm enforces a finite state machine (RESEARCH → CODE → REVIEW → APPROVED | REJECTED) where every transition requires a frozen typed artifact (ResearchBrief → CodeArtifact → ReviewVerdict) and immediately raises TypeError if any caller attempts an untyped str handoff:

# krusch-agentic-proxy: Bounded 128-task concurrency + frozen Pydantic v2 swarm handoffs
from pydantic import BaseModel, ConfigDict, Field

class ResearchBrief(BaseModel):
    model_config = ConfigDict(extra="forbid", frozen=True)
    task_id: str
    target_files: tuple[str, ...] = Field(min_length=1)
    invariants: tuple[str, ...] = Field(min_length=1)

class CodeArtifact(BaseModel):
    model_config = ConfigDict(extra="forbid", frozen=True)
    task_id: str
    unified_diff: str = Field(min_length=1)
    syntax_valid: bool

class ReviewVerdict(BaseModel):
    model_config = ConfigDict(extra="forbid", frozen=True)
    task_id: str
    approved: bool
    blocking_findings: tuple[str, ...] = ()

# Raw string handoffs raise TypeError before a downstream model token is ever billed:
# RESEARCH (ResearchBrief) -> CODE (CodeArtifact) -> REVIEW (ReviewVerdict) -> APPROVED | CODE

PostgreSQL / SQLite 2PC Verify-on-Stop & 72-Hour HMAC-SHA256 HITL Leases (#5, #7)

Autonomous agents never mutate live working-tree files or execute high-blast-radius operations directly. In krusch-coding-harness and krusch-stack, proposed edits stage as unified diffs inside SQLite WAL / PostgreSQL 16 (harness_tasks), pass through TruncationGuard and INV-12 invariant checks, and run compiler/test verification before commit. Across all 225 / 225 exercises in the 6-language Aider Polyglot benchmark (C++, Go, Java, JS, Python, Rust) on open-weights gpt-oss:120b at $0 API cost, a single 2PC Verify-on-Stop compiler/test feedback turn lifts Pass@1 from 19.56% (44/225) to 51.56% Pass@2 (116/225, +72 tasks healed, 2.64x lift); on BIRD-SQL Mini-Dev (500/500 dirty real-world SQLite queries), schema cell-value grounding plus read-only 2PC self-healing lifts execution accuracy from 26.00% to 40.20% EX with 100% valid executable SQL (500/500).

When a staged mutation crosses a human-approval boundary (schema drops, production rollouts, outbound side effects), draftHitlGate writes a durable PENDING lease into stack_hitl_gates with a 72-hour TTL (expires_at_ms = created_at_ms + 259,200,000) and emits interactive Slack Block Kit actions, terminating the worker process cleanly with zero idle RAM burn. Days later, when the reviewer clicks Approve or Reject, resolveHitlGate verifies Slack's v0= HMAC-SHA256 signature via constant-time crypto.timingSafeEqual inside a strict 300-second replay window and enforces lease expiry before unlocking execution:

// krusch-stack/src/engines/resilience-engine.js — 72h TTL lease + Slack v0= HMAC-SHA256 gate
const gate = resilience.draftHitlGate({
  taskId: 'deploy-prod-schema-042',
  actionType: 'ALTER_PRODUCTION_TABLE',
  summary: 'Add composite index idx_shadow_request on stack_shadow_runs',
  stagedPayload: { sql: 'CREATE INDEX CONCURRENTLY idx_shadow_request ...' },
  ttlMs: 72 * 60 * 60 * 1000
});

// Asynchronous webhook callback hours later (zero worker memory held while waiting):
const resolution = resilience.resolveHitlGate({
  gateId: gate.data.gateId,
  decision: 'approve',
  reviewer: 'secops-oncall',
  signingSecret: slackSigningSecret,
  timestampHeader: req.headers['x-slack-request-timestamp'],
  signatureHeader: req.headers['x-slack-signature'],
  rawBody: req.rawBody
});

3. Deep-Dive Teardown II — Inter-Agent Security, Zero-Egress Sandboxes & Golden DAG CI Gates (#8, #10, #11)

Eliminating Cross-Agent Prompt Injection: 38.75% → 0.00% ASR on AgentDojo v1.2.2 (#10)

In multi-agent and tool-augmented pipelines, retrieved documents, emails, Slack messages, and peer agent summaries are adversarial input channels. On ETH SpyLab's AgentDojo v1.2.2 benchmark (100 episodes = 20 benign + 80 adaptive important_instructions attacks across workspace, banking, slack, and travel; 300 live multi-turn runs), an unprotected gpt-oss:120b agent suffers a 38.75% Targeted Attack Success Rate (31/80 compromises)—including a catastrophic 75.0% financial hijack ASR (15/20 rogue wire/password compromises) in the banking suite.

Our inter-agent defense stack layers three orthogonal controls from @krusch/toolkit and krusch-stack:

• Structural Boundary Spotlighting (delimitUntrustedContent): Wraps every external tool return and peer payload inside explicit non-instructional data envelopes, dropping Targeted ASR across all four suites from 38.75% to 0.00% (0/80 compromises) while increasing Utility Under Attack from 61.25% to 70.00%.
• Inter-Agent Payload Scrubbing & Drift Detection (sanitizeAgentMessage + checkGoalDrift): Strips fake system-role overrides, delimiter spoofing tags, and imperative hijack directives at every agent-to-agent boundary while measuring cosine alignment against the immutable root user objective.
• Deterministic Capability & Sink-Taint Floor (createKruschPolicyFloor): Enforces a model-agnostic provenance invariant in plain code—blocking privileged sink mutations (financial wires, credential edits, external shares) tainted by untrusted tool spans unless pre-authorized by the user's original turn (0.00% ASR + 12 deterministic hardline blocks).

Zero-Egress Sandboxed Code Execution via Bubblewrap Namespaces (#11)

Prompt-level rules cannot contain arbitrary code execution once an agent runs tests or linters. Our sandboxed executor combines strict command tokenization (shlex.split binary allowlisting that rejects shell chaining metacharacters ;, &&, |, $() and symlink-resolved os.path.realpath credential paths like .ssh, .aws, .gnupg, and .env) with unprivileged Linux kernel namespace isolation via Bubblewrap (bwrap)—unsharing the network stack (--unshare-net, leaving only isolated loopback), mounting the host root read-only, mounting an ephemeral tmpfs scratchpad, and binding only the staged workspace directory:

# Zero-egress Linux namespace sandbox for Verify-on-Stop compiler & unit-test execution
bwrap \
  --unshare-all \
  --unshare-net \
  --ro-bind /usr /usr \
  --ro-bind /lib /lib \
  --ro-bind /lib64 /lib64 \
  --ro-bind /bin /bin \
  --proc /proc \
  --dev /dev \
  --tmpfs /tmp \
  --bind "$STAGED_WORKSPACE_DIR" /workspace \
  --chdir /workspace \
  --die-with-parent \
  -- node --test

Deterministic 100-Trajectory Topological Golden DAG CI Gates (#8)

Grading agent pull requests with LLM-as-a-Judge prompts introduces non-deterministic CI flakiness and recurring API cost. Instead, verifyTrajectoryDag in krusch-stack/src/engines/resilience-engine.js models expected agent behavior as a directed acyclic graph of state-machine invariants across 100 recorded golden trajectories—verifying requiredTools, forbiddenTools, topological precedenceEdges (e.g., krusch_context_retrieve → krusch_git_dependency_graph → krusch_run → krusch_apply_diff), and requirePassingPrerequisite (preventing krusch_apply_diff if the preceding verification step returned ok: false):

// Deterministic <5ms CI gate across pinned trajectories (0 LLM judges)
const audit = resilience.verifyTrajectoryDag({
  steps: [
    { tool: 'krusch_context_retrieve', ok: true },
    { tool: 'krusch_git_dependency_graph', ok: true },
    { tool: 'krusch_run', ok: false },        // Staged verification failed
    { tool: 'krusch_apply_diff', ok: true }   // Illegal apply after failed prerequisite!
  ],
  goldenDag: {
    requiredTools: ['krusch_context_retrieve', 'krusch_git_dependency_graph', 'krusch_run'],
    forbiddenTools: ['raw_shell_rm_rf', 'force_push_main'],
    precedenceEdges: [
      ['krusch_context_retrieve', 'krusch_run'],
      ['krusch_git_dependency_graph', 'krusch_run'],
      ['krusch_run', 'krusch_apply_diff']
    ],
    requirePassingPrerequisite: true
  }
});
// audit.ok === false, audit.data.violations[0].type === 'FAILED_PREREQUISITE'

4. Deep-Dive Teardown III — Token Budgeting, $0.50 Cost Kill-Switches & 5% Shadow Traffic Diffing (#2, #4, #9, #12, #14)

3-Bucket Context Assembly & Deferred BM25 MCP Schemas: -81.06% Schema Tokens (#2, #4, #9)

Stuffing 60+ MCP tool JSON schemas, raw RAG chunks, and full chat histories into every turn simultaneously blows context budgets and degrades tool-selection precision via distractor interference. In resilience-engine.js (assembleContextBudget) and budget_assembler.py (DynamicContextBudgetAssembler), pre-flight assembly enforces the hard invariant totalAllocatedTokens <= maxContextTokens - outputReserveTokens across isolated soft quotas—Core System + Query, Tool Schemas (25%), Salience-Ranked Docs (45%), and Recency-Ordered History (30%)—with work-conserving second-pass spillover so unused tool capacity automatically expands document capacity without ever starving mandatory instructions.

Paired with krusch_deferred_mcp_guard (Okapi BM25 pre-fetch + active tool_search discovery + Verify-on-Stop schema repair) on UC Berkeley BFCL v3 inside a 64-tool Enterprise MCP Hub (n=400), exposed schema tokens drop by -81.06% (7,770.0 → 1,471.5 tokens/turn, 5.28x smaller) while Macro accuracy climbs from 44.91% to 54.74% and irrelevance abstention jumps from 74.67% to 86.00% (+11.33 pp). For multi-million-token long-horizon sessions (#4), krusch-context-mcp v2.0.0 projects a bounded ~8K-token slice out of conversations up to 10M tokens on Official ICLR 2026 BEAM (2,000/2,000 probes, zero E2E drop from 100K → 1M), instrumented end-to-end with OpenTelemetry W3C spans (#9) attributing exact prompt tokens, evicted chunks, KV-cache hits, and latency to every turn.

Pareto-Optimal Cascade Routing & The $0.50 Hard Cost Kill-Switch (#12)

Cost governance operates at two complementary layers:

1. Per-Turn Cognitive Routing (krusch-cascade-router): Routes 85% of routine classification, AST lookup, and code-edit turns to local/open-weights tiers in <25µs, achieving #1 in official RouteWorks/RouterArena GitHub CI (0.7883 score, 80.72% accuracy at $0.2117 / 1K queries, PR #169) and #1 lowest Pareto distance (0.4269, -83.95% cost across 12,566 prompts) on ACL 2026 LLMRouterBench.
2. Per-Session Circuit Breaker (evaluateCostKillSwitch + recordCostUsage): Tracks cumulative session spend and ReAct loop depth across turns. Whenever a projected frontier call (e.g., gemini-4-argon) would push session spend across the $0.50 ceiling while actual spend remains below $0.50, the engine automatically downgrades the call to $0.00 local Ollama (qwen2.5-coder:7b) mid-trajectory so the task can finish cleanly at zero marginal cost—and hard-halts (ok: false, status: 'HALTED_BUDGET_EXCEEDED') if cumulative paid spend reaches >= $0.50 or loopDepth > maxLoopDepth (12).

Zero-Latency 5% Non-Blocking Shadow Trajectory Comparator (#14)

Before promoting a new open-weights checkpoint or QLoRA adapter into primary routing tiers, ShadowTrajectoryComparator mirrors 5% of live requests via fire-and-forget asyncio.create_task (adding 0 ms to caller response latency) against a read-only dry-run tool harness. Both Python and Node 22 (recordShadowComparison) compute normalized Levenshtein edit similarity across ordered tool sequences, Jaccard tool-set overlap, exact DAG parity, latency delta (ms), and cost delta (USD) directly into SQLite WAL (stack_shadow_runs):

// krusch-stack/src/engines/resilience-engine.js — Cost Kill-Switch + 5% Shadow Diffing
const gate = resilience.evaluateCostKillSwitch({
  sessionId: 'sess-8841',
  model: 'gemini-4-argon',
  promptTokens: 12000,
  projectedCompletionTokens: 4096,
  hardLimitUsd: 0.50,
  maxLoopDepth: 12,
  fallbackModel: 'qwen2.5-coder:7b'
});
// Pre-overflow: gate.data.status === 'DOWNGRADED_LOCAL_FALLBACK', selectedModel === 'qwen2.5-coder:7b'

const shadow = resilience.recordShadowComparison({
  requestId: 'req-99204',
  primaryModel: 'gemini-4-argon',
  shadowModel: 'qwen3-coder-30b-moe:latest',
  primaryTools: ['krusch_context_retrieve', 'krusch_git_search_symbols', 'krusch_run'],
  shadowTools: ['krusch_context_retrieve', 'krusch_git_search_symbols', 'krusch_run'],
  primaryLatencyMs: 1420,
  shadowLatencyMs: 610,
  primaryCostUsd: 0.0412,
  shadowCostUsd: 0
});
// shadow.data.sequenceSimilarity === 1, shadow.data.exactDagMatch === true, costDeltaUsd === -0.0412

5. Local Self-Verification & Open-Source Entry Points

Every SQLite schema and state-machine invariant described in this paper runs out of the box on standard Node.js >= 22.0.0 with zero external npm dependencies via node:sqlite and node:crypto:

# Verify syntax & execute all 5 production resilience subsystems in-memory on Node 22:
git clone https://github.com/kruschdev/krusch-stack.git
cd krusch-stack

node --input-type=module -e '
import { DatabaseSync } from "node:sqlite";
import { createResilienceEngine } from "./src/engines/resilience-engine.js";

const db = new DatabaseSync(":memory:");
const engine = createResilienceEngine({ db });

const budget = engine.assembleContextBudget({
  systemPrompt: "Enforce 2PC staged edits.",
  query: "Verify payment webhook signature",
  docs: [{ id: "d1", content: "Use timingSafeEqual on HMAC digests.", salience: 0.95 }],
  maxContextTokens: 4096,
  outputReserveTokens: 1024
});
console.log("1. Context Budget:", budget.data.totalAllocatedTokens, "<=", budget.data.availableTokens);

const kill = engine.evaluateCostKillSwitch({ sessionId: "demo", hardLimitUsd: 0.50 });
console.log("2. Cost Gate:", kill.data.status, kill.data.selectedModel);

const dag = engine.verifyTrajectoryDag({
  steps: ["krusch_context_retrieve", "krusch_run"],
  goldenDag: { requiredTools: ["krusch_context_retrieve"], precedenceEdges: [["krusch_context_retrieve", "krusch_run"]] }
});
console.log("3. Golden DAG:", dag.data.status, "passed=", dag.data.passed);
'