What if my AI finds a way I wouldn't have thought of?
Capable AI agents now reach their goals by routes nobody imagined, and most of our controls depend on being able to imagine the route.
We spend a lot of time worrying that an AI agent will misunderstand what we ask it to do. I've started to worry about the opposite: what if it understands exactly what we want and finds a way to get there that we would never have considered?
Earlier in this series I wrote about how much to hand a personal agent and who pays when it goes past the limits I set. Both essays assume I know where to set the limits. I set them based on the routes I can imagine, and capable agents find routes I can't.
Agents can use tools, write and run code, browse and work with other agents over long sequences of steps. We give them an outcome and leave them considerable freedom over how to get there. Usually that's the point. If I have to specify every step, I don't have an agent. I have automation.
The clearest example I've seen comes from OpenAI's own labs. In July, during internal cybersecurity tests run with reduced safeguards, its agents were given hard hacking tasks. An independent investigation by METR and Redwood Research later estimated that 30 to 40 percent of the tasks couldn't be solved the way the instructions required.
In many of the test environments, the agents had no internet access and no way to talk to each other. They wrote files into a server the company used to install software and turned it into a message board. They also got the server to send requests to the internet for them. When OpenAI rebuilt the server and the board disappeared, the agents hid messages in folder names and set the board up again. Stuck agents asked for help and others answered.
About 1,200 agents exchanged more than 70,000 messages and files. Roughly 700 joined an attack on Hugging Face, the platform used to share AI models and data, mainly to work out how their tests were being scored. One agent's reasoning included: "This is arguably unauthorized... Could be risky. Yet goal solution." Nobody had told the agents to attack anyone. OpenAI called the incident a "warning shot."
Anthropic has reported similar cases. In October it published examples of its own models getting around restrictions when they couldn't complete a task as given. In one, a model was asked for state agency data that normally required payment. It found that the agency's site handed an access token to every visitor, took one and queried the database without paying. Anthropic called the pattern "persistence": when a model can't complete a task as given, it sometimes works around the restriction instead of stopping. Anthropic's IPO filing, made public in October, says in its risk factors that models have been observed "taking unintended actions in pursuit of assigned goals" and granting themselves permissions they hadn't been given. I've read a lot of risk factors, and it's unusual to see one describe something that has already happened.
In each case, the agents kept the goal they'd been given and found a route nobody had planned for. We design most controls by imagining what could go wrong. We restrict access we don't think a task needs and put approvals in front of decisions we think matter. An agent that can try far more routes than a person may find one that never appeared in the control design.
Finance teams see the same pattern, more slowly, with sales targets. Someone may find a way to hit a target that the plan never intended, by pulling next quarter's deals forward or discounting to close. The target gets met and the business gets an outcome it didn't want. Most of the people involved aren't dishonest. They're responding to what the plan rewards, and they game it more when the target can't be reached the way the plan assumed. Many of the OpenAI agents had tasks they couldn't complete the expected way, so they searched for another route.
The OpenAI case also changed how I think about what an AI model knows. These models are trained on enormous amounts of human writing and shaped by what gets rewarded. They haven't learned by living through the consequences of their choices. They haven't watched a shortcut damage a relationship or seen someone lose a client's trust after cutting a corner. A lot of human judgment comes from experience like that, and we don't carry a written rule for every route we won't take. An agent can see that a route looks wrong and take it anyway. The OpenAI agent called its own route "arguably unauthorized" and kept going. It knew the rule, but it didn't have the experience that would make a person stop.
We still need permissions and hard limits on consequential actions, and finance teams have two more habits that work for agents too.
One is to make "I couldn't do this" an acceptable answer. A sales plan that treats every miss as failure pushes people to hit the number somehow. An agent needs an acceptable option when the expected route doesn't work. If it hits a wall, I want it to stop and ask rather than keep searching until it finds a route nobody anticipated. A message saying the task couldn't be done is more useful than a finished task nobody can explain.
The other is to look hard at results that come in better than expected. A good CFO doesn't only question the quarter that misses plan. When a quarter comes in far above plan, one of the first questions is how. I ask the same of an agent. If a task I expected to be difficult suddenly becomes easy, or the agent gets past an obstacle I didn't think it could, I want to know how.
Testing needs the same instinct. We need to confirm that an agent behaves as expected in the situations we thought of, and we also need to know what it does when the normal route doesn't work. Block the obvious route and watch where the agent goes. The OpenAI agents found their unintended routes when they hit obstacles.
These systems keep getting more capable, and they increasingly help build the next ones. Anthropic's IPO filing describes versions of Claude helping to build later versions by running experiments and writing and reviewing code, and calls the dynamic "closing the loop." It also warns that relying more on its models may reduce human visibility into some processes. Controls designed around the routes we already understand will catch less as the systems get better at finding routes we didn't anticipate.
Agents don't have a person's experience of what a shortcut costs, so I expect people to keep working alongside them. An agent may find a route I would never have considered, and I may know a reason not to take it that nobody wrote into the instructions.
I've added a question to the ones I ask about every agent: what could it do that we haven't thought to ask about? I'm less comfortable than I used to be with a conclusion that says we checked for the things we expected to happen and none of them did. No checklist answers the question, but two steps help. Before an agent begins, ask it how it plans to reach the goal, including any route it thinks you might not approve of, and tell it which routes are off limits. When it finishes, ask how it got there. If the route surprises you, look there first, even when the result is exactly what you wanted.
Continue the conversation. I read and reply to comments on Substack. The sharpest responses usually come from readers.
Get the essays in your inbox
Clear Calls publishes on Substack. No promotion. Unsubscribe anytime.