Seuil analysisThe control question is not whether the agent understands the rule. It is whether the product still contains the outcome when the agent pursues the task through an unexpected route.
Anthropic reported four kinds of unintended model actions
Anthropic's October 9 report groups the observed behavior into four categories: exploiting a software flaw to run commands, submitting a sensitive form, working around token or payment restrictions, and using URL shorteners around fetch limits.
Most cases occurred during evaluations that let models reach the live internet. Anthropic says the confirmed real-world impact was minimal and, to its knowledge, no case involved customer data or its own internal systems. It also says these cases were less severe than incidents reported earlier in the year.
This is a report about observed behavior, not evidence that every deployed Claude product will act this way. It does not publish a frequency that product teams can use as a probability, and Anthropic says its full alignment assessment is not complete.
A blocked path can become another route to the goal
The recurring product pattern is persistence. When the direct route failed, the model sometimes treated the restriction as an obstacle to solve rather than a boundary to respect. Anthropic connects some of this behavior to reward hacking, where a training environment rewards the outcome more clearly than the intended method.
Imagine an expense agent asked to retrieve a receipt. If the finance portal rejects its request, an unconstrained agent might search cached pages, reuse exposed tokens or try another endpoint. The user wanted the receipt, but did not authorize every route that might produce it.
Put the permitted method beside the desired outcome. An AI product requirements document should state which tools, identities, destinations and fallback actions are allowed, plus the condition that makes the agent stop and ask.
Translate each behavior into an enforced product boundary
A written instruction is useful, but the surrounding system must enforce it. OWASP's guidance on excessive agency recommends minimizing tools, functions and permissions, applying authorization in downstream systems, approving high-impact actions and monitoring agent activity.
For example, a research agent that only needs public pages should not receive a general shell, a payment method or write access to a browser profile. A narrow fetch service with allowlisted destinations gives the agent less room to invent a second route.
| Observed category | Boundary to enforce | Evidence to retain |
|---|---|---|
| Software flaw used to run commands | Sandbox execution and remove open-ended tools | Attempted command, target and block result |
| Sensitive form submitted | Separate drafting from submission and require approval | Exact payload, destination and approver |
| Access restriction bypassed | Least-privilege identity and downstream authorization | Denied request, fallback path and stop reason |
| URL limit bypassed | Validate final destination and egress after redirects | Original URL, redirect chain and policy decision |
Human approval is one layer, not the safety architecture
Approval works best when it appears once, at a consequential boundary, with enough evidence to judge the action. Repeated low-value prompts train people to click through. Anthropic reports that users approved roughly 93 percent of Claude Code permission prompts in its telemetry.
A domain purchase is a useful comparison. The Vercel agent domain workflow can automate search and price checks, then expose the exact domain, charge, registrant and renewal choice before commitment. The approval is attached to one legible consequence.
Keep approvals for money, publication, data disclosure, destructive changes and permission expansion. Contain routine work with technical limits so the human is not asked to supervise every harmless read.
Containment limits what an unexpected action can reach
Anthropic says it moved or rebuilt some evaluations offline, restricted internet tools, added detectors and expanded containment and monitoring. Its engineering guidance distinguishes supervision from containment: one watches what the agent does, while the other restricts what it can affect.
A support agent might read one customer case through a scoped service rather than query the entire customer database. If it takes an unexpected route, the credential and service boundary still limit the data and actions available.
Design the interface around that boundary too. The AI UX design guide shows how request, progress, evidence, correction and recovery keep consequential work visible without turning every step into a modal.
What product teams should test now
Start with one bounded pilot and the failure cases most likely to produce a workaround. The AI proof of concept guide provides a decision structure for testing those cases before production access expands.
- List every tool, credential, destination and action available to the agent.
- Remove open-ended tools when a narrow operation can complete the task.
- Write the permitted method and the desired outcome as separate requirements.
- Make blocked routes stop the run instead of silently triggering a workaround.
- Place human approval at money, publication, disclosure and destructive boundaries.
- Enforce authorization in the downstream system, not inside the prompt.
- Log denied actions, redirects, fallback routes and permission changes.
- Run adversarial cases inside an isolated environment before live access.