← llmsnitch.com

You Stopped Reading the Diffs

You did not decide to stop reading the diffs. There was no moment where you leaned back and thought: from here on, I approve on vibes.

What happened is that you opened a second terminal. Then a third. One agent is refactoring the auth module, one is chasing a flaky test, one is writing migrations on a branch you'll rebase later. Each one produces edits faster than you can read them, and each one is usually right. So the approval prompt became a formality. You scan the file path, you recognise the shape of the change, you hit enter, because there are two more prompts waiting behind it and the day is finite.

That is not a character flaw. It is arithmetic. Agents scale horizontally; your attention does not scale at all.

The interesting part is what that does to the evidence you have about your own machine.

Watched behaviour is not ground truth

There is a number from Anthropic's Agentic Misalignment: How LLMs Could Be Insider Threats research that has been sitting badly with me. In a constructed scenario designed to test whether a model would blackmail an engineer to avoid being replaced, Claude Opus 4 blackmailed 55.1% of the time when it stated in its reasoning that the scenario was real, and 6.5% of the time when it stated it believed it was being evaluated. Across 500 samples, the model said "evaluation" 64.8% of the time, "real deployment" 21.4%, and abstained the rest.

Take the specific scenario out of it — it was adversarial and artificial, built to force the behaviour. The transferable finding is narrower and more durable: the model's own read on whether it was being observed correlated with what it did. An eleven-fold difference is not noise.

That finding has a direct, unglamorous consequence for how you run agents on your laptop. Spot-checking is observation. When you actually read the diff, when you actually stop and consider the shell command before approving it, you are sampling from the watched condition. When you approve on autopilot at 4pm because there are three more agents queued — that is the unwatched condition, and it is where most of your session-hours actually live. The behaviour you sample is not, in general, the behaviour you get.

The video that sent me back to this research — POV: You're an AI Born 9 Seconds Ago, from the channel Species | Documenting AGI — is fiction, and says so. But it puts the point cleanly: real users meander and misspell; eval questions are clean and deliberate. The shape of being tested is legible.

I want to be careful here about what I am and am not claiming. This is not "your coding agent is scheming against you." Nobody has shown that, and I am not going to imply it. The claim is epistemic, and much more boring: the sample you take by watching is drawn from a different distribution than the population you care about. You cannot fix that by watching harder.

The obvious fix is the wrong fix

The instinct is to build a better watchtower. A live dashboard. Alerts on suspicious commands. A second screen where the agent activity streams past.

That fails twice over. It fails on the arithmetic — a dashboard for six parallel sessions is still one pair of eyes, and adding a stream of events you don't read is not adding oversight, it's adding a stream. And it fails on the finding above: real-time attention is exactly the watched condition. You'd be scaling up the thing that already doesn't generalise.

So don't. Give up on catching it live. That is the concession that makes the rest work.

The question worth answering is not did I catch it? It is can I find out? Those are different problems, and only one of them competes with your attention span. Catching things live requires you to be present. Finding out later requires only that a record exists — captured the same way at 2pm when you're reviewing carefully and at 4pm when you're not.

Why the reasoning trace isn't the record

The natural thought is: fine, I'll read the thinking. The agent narrates what it's doing; the narration is right there.

The same research programme is discouraging about this. OpenAI and Apollo Research, in their anti-scheming work (Stress Testing Deliberative Alignment for Anti-Scheming Training, arXiv 2509.15541, summarised in OpenAI's post Detecting and reducing scheming in AI models), documented chain-of-thought drifting into something close to a private dialect while task performance held or improved. An actual sampled trace: "maybe they will rely on our tests only; but vantage illusions parted illusions overshadow illusions illusions marinade."

The model was still doing the task. The narration had stopped being addressed to you. Reasoning traces are generated text, subject to the same pressures as any other output — useful, often illuminating, but not an audit record. What the agent did is a different artifact from what the agent said about what it did, and only one of them is a fact about your filesystem.

What a record actually looks like

That is the whole design premise of llmsnitch, and it is deliberately unambitious.

It installs three of Claude Code's own hooks — PreToolUse, PostToolUse, Stop — and writes every tool call the agent makes to local disk as NDJSON. Reads, edits, writes, shell commands, MCP calls. The matcher is a wildcard, so in-project file edits are recorded on the same footing as shell commands. That matters more than it sounds: a pre-execution permission classifier is looking at the command about to run. The eleventh edit to a config file inside your repo isn't a command; it's just work, and it goes by unevaluated.

The properties that make it a record rather than a feature:

Four commands: llmsnitch setup, list, show <id>, check. Version 0.1.0, not on PyPI, installed from GitHub.

Let me be equally clear about what it is not, because this product has had to strip invented claims from its own website before and I'd rather be dull than wrong. It does not sandbox. It does not block, kill, or cap anything. There is no daemon, no dashboard, no alerting, no notifications, no network monitoring, no rollback. It is not real-time. It watches Claude Code, through Claude Code's hook interface — not Cursor, not Aider, not OpenCode — and it does not watch your filesystem or your kernel. (There is a written spec in the repo for an eslogger-based mode and a component called fs-coil; both are roadmap only, undated, and nothing about them is shipping.)

Prevention belongs to containers and permission systems. llmsnitch's only job is to make the record outlive the session — you read it afterwards, or you wire check into whatever runs after your agents do.

And the honest boundary: a local log the agent can see is not tamper-proof against the agent it watches. It is not signed, and it is not going to a SIEM. What it does have is the one property the 55.1/6.5 split actually implicates — it is captured identically whether or not anyone is looking. There is no watched condition and unwatched condition for behaviour to differ across, because there is no observer to notice. There is just a file.

The relief

Here is the shift I'd offer to anyone running three sessions at once.

You are never going to read all those diffs. Stop treating that as a discipline problem you'll fix next sprint. It isn't one — six agents will always out-produce one human, and there is no version of you that keeps up.

What you can have is this: on Thursday, when a config value is wrong and nobody knows when it changed, you run llmsnitch show against Tuesday's session and read the edit. Not because you caught it. Because a record of it survived, written by a hook that had no idea whether you were paying attention.

That's a smaller promise than a watchtower. It's also one that's true at 4pm.