I entered the SANS FIND EVIL! hackathon with an agent that catches its own hallucinated indicators. It didn't win. Reading the entries that placed, the ones I looked at had reached for the same defence I had — check the first language model with a second language model. But a model can be argued out of a correct refutation. So I rebuilt the whole thing around an oracle that cannot be argued with: a finding is admitted only when pure Python re-reads the cited tool output and confirms the field really says what the claim says it says. 1,083 scored claims, 119 crafted attacks, one command to run it.

Every security and GRC team rebuilds the same pipeline one-off: chunk the SOC 2s and policies, embed them, retrieve per question, prompt an LLM, paste into the form. Done naively it hallucinates confident answers with no supporting evidence — the single worst failure mode in compliance. attestq is the small, model-/embedder-/store-agnostic kernel I extracted from two production tools (a vendor due-diligence assistant and a customer-assurance drafter) that bakes in the two fixes that actually matter: a confidence gate that answers "insufficient evidence" without ever calling the LLM, and a wide rerank window so the one document that had the answer doesn't get dropped. pip install attestq.

The first version of our AI SOC had eight roles and zero memory — every alert started from scratch. This post adds the layer that makes a SOC senior instead of just fast: an event-sourced case memory. Agents recall similar prior cases as precedent, every verdict is scored against ground truth so we can prove whether the agents are actually right, any past decision can be interrogated from the recorded trace instead of a hallucinated rationale, and cross-incident campaigns surface that no single analyst working one ticket would ever see. Plus the honest part — we flag-gated precedent so we can A/B whether memory actually helps.

A tool-calling agent burns almost all its wall-clock on the loop — decide a call, read the result, decide the next. Writing the final answer is the part that wants a strong model. They don't have to be the same model. langchain-failover 0.2 adds TieredChatAgent — run the gathering loop on a cheap/local model, compose the answer on a frontier one. On a contended local GPU that turned a multi-minute final turn into a couple of seconds.

detflow could already go from plain English to a detection rule. v0.2 adds the reverse direction — paste a CVE advisory or a CTI writeup and get back tactic-ordered ATT&CK techniques, generated Sigma + YARA + Suricata (Sigma linted in place), and one-click STIX 2.1 / ATT&CK Navigator / Markdown-brief exports. Same never-raises, model-agnostic, untrusted-input-hardened design.

My SANS FIND EVIL! hackathon entry. Protocol SIFT turns the SIFT Workstation into a Claude Code agent over 200+ forensic tools — and it hallucinates indicators because it eyeballs 50,000-line tool dumps. find-evil bolts my open-source iocflow onto it as an MCP server plus one skill, so the agent stops guessing and reconciles its own findings against a deterministic, FP-defended extractor. On the real SANS compromised image it found fileless PowerShell Empire C2, corrected its own "no external C2" read, and refused to invent attribution on the clean host.

A multi-agent AI SOC stopped posting verdicts — no errors, no crash, no alert. Just silence. The failover had worked perfectly, and that was exactly the problem. A post-mortem on success-shaped failures in LLM systems, and why "it didn't throw" is the most dangerous sentence in production AI.

I kept rebuilding the same four things inside every detection-as-code pipeline — lint a rule, draft one from plain English, check it against what you already run, and review it like a senior engineer. So I extracted them into detflow, a vendor-neutral OSS Python package. Deterministic lint and overlap with no dependencies, model-agnostic drafting and review, and a never-raises contract so it degrades instead of breaking.

After building SOC-in-a-Box — a multi-agent AI SOC where one local LLM wears every hat behind a human-in-the-loop gate — I distilled the durable lesson into iocflow, an open-source Python package for the whole IOC lifecycle. Deterministic primitives (extract → enrich → comment → hunt → block) as tools, a LangGraph multi-agent team on top, and three-layer authority so the LLM never gets the final say on a destructive action.

An AVP-sponsored multi-agent SOC where one local LLM plays Sentinel, Tier 2, IR Lead, Threat Intel, SOC Manager, Detection Engineer, and Threat Hunter — coordinated over a Redis Streams bus with a human-in-the-loop approval gate before any real-system action. The framework choices, the architectural trade-offs, and the backtest harness that lets us put real numbers on agent quality before going live.