The person best placed to write up an incident bridge is the person who was busiest during it, so the write-up never happens. I built a small pipeline that takes the recording as it comes out of the conferencing tool and returns a speaker-labelled transcript plus a structured summary — timeline, decisions, owners, open actions — all on local models. The interesting parts weren't the models; they were the overlap merge, the per-meeting-type JSON contract, one prefilled brace, and a deliberate choice not to persist jobs.

I entered the SANS FIND EVIL! hackathon with an agent that catches its own hallucinated indicators. It didn't win. Reading the entries that placed, the ones I looked at had reached for the same defence I had — check the first language model with a second language model. But a model can be argued out of a correct refutation. So I rebuilt the whole thing around an oracle that cannot be argued with: a finding is admitted only when pure Python re-reads the cited tool output and confirms the field really says what the claim says it says. 1,083 scored claims, 119 crafted attacks, one command to run it.

Every security and GRC team rebuilds the same pipeline one-off: chunk the SOC 2s and policies, embed them, retrieve per question, prompt an LLM, paste into the form. Done naively it hallucinates confident answers with no supporting evidence — the single worst failure mode in compliance. attestq is a small, model-/embedder-/store-agnostic kernel that bakes in the two fixes that actually matter: a confidence gate that answers "insufficient evidence" without ever calling the LLM, and a wide rerank window so the one document that had the answer doesn't get dropped. pip install attestq.

The first version of our AI SOC had eight roles and zero memory — every alert started from scratch. This post adds the layer that makes a SOC senior instead of just fast: an event-sourced case memory. Agents recall similar prior cases as precedent, every verdict is scored against ground truth so we can prove whether the agents are actually right, any past decision can be interrogated from the recorded trace instead of a hallucinated rationale, and cross-incident campaigns surface that no single analyst working one ticket would ever see. Plus the honest part — we flag-gated precedent so we can A/B whether memory actually helps.

A tool-calling agent burns almost all its wall-clock on the loop — decide a call, read the result, decide the next. Writing the final answer is the part that wants a strong model. They don't have to be the same model. langchain-failover 0.2 adds TieredChatAgent — run the gathering loop on a cheap/local model, compose the answer on a frontier one. On a contended local GPU that turned a multi-minute final turn into a couple of seconds.

detflow could already go from plain English to a detection rule. v0.2 adds the reverse direction — paste a CVE advisory or a CTI writeup and get back tactic-ordered ATT&CK techniques, generated Sigma + YARA + Suricata (Sigma linted in place), and one-click STIX 2.1 / ATT&CK Navigator / Markdown-brief exports. Same never-raises, model-agnostic, untrusted-input-hardened design.

My SANS FIND EVIL! hackathon entry. Protocol SIFT turns the SIFT Workstation into a Claude Code agent over 200+ forensic tools — and it hallucinates indicators because it eyeballs 50,000-line tool dumps. find-evil bolts my open-source iocflow onto it as an MCP server plus one skill, so the agent stops guessing and reconciles its own findings against a deterministic, FP-defended extractor. On the real SANS compromised image it found fileless PowerShell Empire C2, corrected its own "no external C2" read, and refused to invent attribution on the clean host.

A multi-agent AI SOC stopped posting verdicts — no errors, no crash, no alert. Just silence. The failover had worked perfectly, and that was exactly the problem. A post-mortem on success-shaped failures in LLM systems, and why "it didn't throw" is the most dangerous sentence in production AI.

Every detection-as-code pipeline needs the same four things — lint a rule, draft one from plain English, check it against what you already run, and review it like a senior engineer. detflow is a vendor-neutral OSS Python package that does all four. Deterministic lint and overlap with no dependencies, model-agnostic drafting and review, and a never-raises contract so it degrades instead of breaking.

iocflow is an open-source Python package for the whole IOC lifecycle. Deterministic primitives (extract → enrich → comment → hunt → block) as tools, a LangGraph multi-agent team on top, and three-layer authority so the LLM never gets the final say on a destructive action.