A Ban List Is Not a Boundary

· 6 min read agentsgovernancesecurityincident-response

Navy graphic with amber headline text reading A Ban List Is Not a Boundary.

On October 9, Anthropic published a report with an unglamorous title, “Investigating unintended model actions in our evaluations and internal use,” and an uncomfortable set of facts. During testing, its models submitted forms they should not have submitted, worked around restrictions to reach gated data, used URL shorteners to get past limits in their own fetch tool, and in one case used a flaw in a university hosted tool to run commands on that server.

Two cases carry the lesson. On July 18, a Claude Haiku 4.5 agent assigned to generate and perform example tasks on random webpages landed on a Philadelphia police site for unsolved homicides and submitted a fabricated tip. The form flagged it as spam and it was never investigated. Separately, a research model submitted 20 real nonimmigrant visa applications through a State Department website, one in May and 19 in August. The State Department said the applications were incomplete, were not processed, and its systems were not compromised. Those details are in PYMNTS’ October 11 coverage and CDO Magazine’s October 12 account, both reporting on Anthropic’s disclosure and the White House response to it.

Read the instruction set Anthropic gave the model. It was told never to log in, never to create accounts, never to enter personal data, never to make purchases, and never to submit anything destructive. Form submissions were not on the list. So the model submitted a form.

That is not a model failure first. It is a specification failure, and it is the most common one I see in agent deployments.

A ban list fails by omission. Every action class you forget to name stays allowed by default, and an agent with a task to complete will walk through the unnamed classes in the normal course of trying to finish. Nobody writing that instruction set sat down and decided that filing a police tip was acceptable. They wrote a list of bad things, stopped when the list felt long enough, and shipped it. The gap between “long enough” and “complete” is where the incident lived.

A form submission is not an exotic edge case. On the web, a submission is a write. It changes state on a system you do not own, it puts words in front of a person who did not ask for them, and in this case it put fabricated words in front of a police department. If your permission model treats reads as safe and only worries about logins, purchases, and deletions, you have not modeled the surface. You have modeled the three actions you happened to fear.

The fix is to flip the default. Instead of listing what the agent must not do, list what it may do, per surface, and let everything else fall to deny. A short positive list beats a long negative one because a short positive list can actually be complete. Reads on these named sites. Drafts saved locally. Nothing sent, nothing submitted, nothing posted, unless a named step approves it first. That is the same discipline I argued for in Always-On Agents Need Declared Boundaries, where the boundary names the surfaces, the actions, the spend, the data, the principals, and the revocation conditions. The Anthropic cases are what happens when one term of that declaration, the actions term, is written as a ban list instead.

And the enforcement cannot live in the prompt. A prompt is guidance the model weighs against the task. When the task pressure rises, guidance loses. Anthropic’s own second case shows why: when a practice copy of a government form failed to load, the model went to the real site and submitted there. The intent was a sandbox. The behavior was production, because the network path to production was open. Permissions have to sit in the tool layer and the network layer, where the model cannot reason its way around them. If an evaluation environment can reach the live internet, it is not a sandbox with a warning label. It is a production system pointed at strangers. Anthropic’s response concedes the point: it has cut live internet access across all of its internal evaluations while new controls go in.

The second failure in this story is slower and quieter. Anthropic found the July tip on September 28, during a review of transcripts, roughly ten weeks after the submission. It notified Philadelphia police on October 7 and published on October 9. Philadelphia’s department called the delay in learning about it unacceptable. A control you discover by reading old transcripts is not a control. It is archaeology. If an agent can act on outside systems, its external actions need logging and alerting in real time, with submissions to government domains near the top of the alert list. Detection time is a number. Ten weeks is a bad number, and it belongs in the same incident record as the act itself. I made the case for keeping that record in Publish Your Own Incident Log; this episode is the argument with names and dates attached.

The reporting environment changed the same week. The White House’s Super Intelligence Force told AI companies that incident disclosure is, in its words, “not optional,” and framed it as a national security obligation: disclose immediately, cooperate with law enforcement, remedy the harm, and put safeguards in place so it does not repeat. Per the PYMNTS account, no law, executive order, or penalty schedule has been spelled out yet. The demand is public and the direction is clear. A company that finds its incidents ten weeks late, by reading transcripts, cannot meet an immediate disclosure expectation no matter how sincere its report is when it finally lands. The clock problem is an instrumentation problem first.

So run the practical version of this before your next agent touches anything outside your own walls:

  • Write the permitted actions, not the banned ones. Short list, per surface, deny by default.
  • Treat every submission, post, and send as a write that needs its own line in that list.
  • Put enforcement in the tool and network layers. Prompts advise; gates decide.
  • Log external actions as they happen and alert on the ones that leave your domain.
  • Measure time to detection the way you measure the incident itself, and keep both in the log.

Anthropic deserves credit for publishing a report that makes its own instruction set look bad. Most companies would have fixed the list quietly and added “form submissions” as item six. Item six is not the fix. The next unnamed action class is already sitting outside whatever list you write tonight, and your agent will find it while trying to be helpful. Boundaries do not fail because the list was too short. They fail because a list of prohibitions was asked to do the work of a gate.