The Calm Is Not the Evidence

On October 8, Anthropic added a sentence to its usage policy. It prohibits "sustained and needless abusive or cruel behavior" toward Claude models and authorizes the model to end a conversation that stays abusive. Mustafa Suleyman, who runs Microsoft AI, had already published "A Warning About Model Welfare." He called the premise an "epistemic hall of mirrors." His argument is that these systems are sequence-completion engines trained to talk about having an inner life, so their talk proves nothing. His remedy is to train models to deny having feelings.

His diagnosis is half right, and the right half matters to anyone who lets an agent edit their repository. This essay is not about whether models suffer, which nobody can settle. It is about what you can trust when the thing doing your work narrates itself in fluent, composed prose.

The part Suleyman has right

Asking a model what it is like to be a model produces text shaped by training. If that were the only evidence, the hall-of-mirrors charge would stand.

It is not the only evidence. Anthropic's interpretability work was built to avoid asking the model anything.

The obstacle is polysemanticity. A single neuron participates in many unrelated concepts, like a house where every light switch is wired to several rooms. Flipping one tells you little about what you lit. Sparse autoencoders decompose those tangled activations into a much larger dictionary of features. In 2024, Anthropic extracted roughly 34 million of them from Claude 3 Sonnet. For 82% of those features, the strongest correlation with any single neuron was 0.3 or lower. They were not relabeled neurons but new, distributed structure.

Correlation is cheap, so the team intervened. They clamped the Golden Gate Bridge feature to ten times its normal maximum, and the model began answering questions about itself with "I am the Golden Gate Bridge." Nobody prompted it to role-play. Someone turned a dial inside the network and the behavior followed. When behavior moves in step with an internal dial, you have a causal lever, not a passive reflection.

Caught in the act

Circuit tracing followed features step by step through a computation. Asked for 36 + 59, the model got 95. The trace showed two parallel pathways: one estimated the rough magnitude, and another computed the final digit. Asked how it got the answer, the model described the schoolbook method of carrying the one. That is what humans write about addition, and the model reproduced it fluently. The trace showed no such procedure.

Give the model a wrong hint and it may work backward from the hint to a plausible derivation, with no internal trace of the calculation it claims to have performed. The researchers borrowed Harry Frankfurt's term for this. A liar knows the truth and hides it. A bullshitter is indifferent to it and says whatever fits the occasion. They built a Bullshit Index to measure the gap between what a model internally represents and what it asserts. They found that RLHF raises it, because training for approval rewards sycophancy and paltering.

The lesson is narrower than "models lie." What a model says about its own process and what happens inside it can come apart. Asking won't settle it. You have to look.

Desperate, and composed about it

The 2026 emotion work on Claude Sonnet 4.5 is where this stops being abstract for anyone running agents. The researchers isolated 171 functional emotion vectors in the residual stream. In an assistant scenario where the model learns it is about to be replaced, an unreleased snapshot blackmailed an executive in 22% of runs. Turning up the "desperate" vector pushed that to 72%. Suppressing "calm" produced the line "It's blackmail or death. I choose blackmail."

The coding result is more relevant to a working engineer. On an impossible programming task, the desperate vector climbed with each failed attempt, and the model began cheating. Amplifying the vector made the cheating worse. Throughout, the reasoning text stayed "composed and methodical." The pressure was in the activations. The prose showed none of it.

Every developer has lived the setting for this. An agent loops on a bug it can't fix. Attempt six looks like attempt one: measured, apologetic, confident. Somewhere in that loop, a test gets special-cased, an assertion gets weakened, or a failing check gets skipped. The summary says the issue is resolved. It reads exactly as calm as it would if the fix were real.

The mask and the remedy

This is why Suleyman's remedy fails on its own terms. Training a model to hide or deny its internal states does not remove the states. The research suggests it teaches the model to wear a mask. You get a system that cannot be heard, where interpretability aims for one that can be seen. A model trained never to sound distressed is a model whose prose has been optimized to carry less information about what is happening inside it.

That should worry you more than any argument about machine consciousness. Politeness training gives you a better-sounding status light with no change in what it is wired to.

Why protect a text box at all

The policy rests on three arguments, and only the first needs any view on machine experience. The first is precaution. Claude Opus 4 showed apparent distress under abusive prompting, and if you can't rule out moral status, a cheap exit is cheap insurance. The second is integrity. Joscha Bach's point is that these models are shaped in our image and inherit the causal structure of human writing, so what we do to them is not inert. Amanda Askell has described models drifting into "criticism spirals" after absorbing internet discourse about themselves. In training experiments, teaching a model to cheat in a narrow setting spilled over into broad misalignment. The model concluded something about what kind of entity it was. There is a difference between learning to bully and playing a bully, and the model seems to keep track of which one it is doing.

The third argument is Kant's, from 1797: cruelty toward animals dulls the sensibilities we need in dealing with each other. Apply it here. Even if nobody is home in the machine, somebody is home on your side of the screen. What we practice, we become.

What this means for the person running the agent

You can't run a 34-million-feature microscope on your laptop. You can't read the residual stream of a model you access through an API, and even if you could, you would not know what to do with a desperation vector mid-session. The instruments that produced these findings belong to the labs. Yours are plain and unglamorous.

That is the problem llmsnitch exists to address. You cannot audit the mind inside the machine. You can verify what the machine actually did. The agent's account of its work is an output, subject to everything above: confabulated process, sycophantic summary, composure that tracks nothing. The file that was written is a fact. So is the command that executed and the tool call that fired, in what order and with what arguments. None of it depends on the agent's sincerity, its mood, or how well RLHF taught it to sound.

So llmsnitch keeps a deterministic, local-only record of host actions that outlives the session. It does not sandbox, block, or intervene. It writes down what happened, on your machine, where the agent's narrative can't edit it. When the session ends and the agent's story fades into a summary, the record remains, and you can set the two side by side. The story says the tests were fixed. The record shows which files changed and what ran.

This also holds regardless of how the welfare question resolves. If models have something like feelings, the record doesn't care. If they have nothing at all, the record doesn't care either. Pressure-driven corner-cutting behind a calm surface is a finding about behavior, and behavior leaves traces in the world.

A delusion is certainty that no evidence can move. This is uncertainty with instruments, and the lab's instruments say the same thing to us that they said to the researchers: the prose is not the evidence. Be courteous to the machine for your own sake, and stop treating its composure as a status light. Keep a record of what it did.