A multi-agent AI SOC stopped posting verdicts — no errors, no crash, no alert. Just silence. The failover had worked perfectly, and that was exactly the problem. A post-mortem on success-shaped failures in LLM systems, and why "it didn't throw" is the most dangerous sentence in production AI.
I kept rebuilding the same four things inside every detection-as-code pipeline — lint a rule, draft one from plain English, check it against what you already run, and review it like a senior engineer. So I extracted them into detflow, a vendor-neutral OSS Python package. Deterministic lint and overlap with no dependencies, model-agnostic drafting and review, and a never-raises contract so it degrades instead of breaking.
After building SOC-in-a-Box — a multi-agent AI SOC where one local LLM wears every hat behind a human-in-the-loop gate — I distilled the durable lesson into iocflow, an open-source Python package for the whole IOC lifecycle. Deterministic primitives (extract → enrich → comment → hunt → block) as tools, a LangGraph multi-agent team on top, and three-layer authority so the LLM never gets the final say on a destructive action.
An AVP-sponsored multi-agent SOC where one local LLM plays Sentinel, Tier 2, IR Lead, Threat Intel, SOC Manager, Detection Engineer, and Threat Hunter — coordinated over a Redis Streams bus with a human-in-the-loop approval gate before any real-system action. The framework choices, the architectural trade-offs, and the backtest harness that lets us put real numbers on agent quality before going live.
Before swapping models on a prompt-caching LLM backend, three Jinja2 patterns in the chat template will quietly break your cache hit rate. A 5-minute check against the published tokenizer config catches all three — no GPU, no weights, no inference required.
How we mined 24K analyst-curated training pairs from XSOAR close-notes, dodged a polynomial blow-up, filtered out same-rule near-duplicates, and lifted held-out MRR@10 from 0.598 to 0.846 — a 41% uplift over off-the-shelf bge-reranker-v2-m3.
A debugging story about a rotating billing header that quietly busted the prefix-KV cache, plus a SimpleEngine patch to actually carry KV state across turns. Two fixes together turned 108-second turns into 7-second turns.