McKinsey's State of AI survey, published in August 2026, shows how uneven progress is. Among organizations earning more than $1 billion annually, 40% are scaling AI agents in at least one business function, up from 27% last year. For smaller organizations, the rate remains at 22%. Yet only about 20% of respondents say their organizations are scaling AI agents across the business.
It's easy to blame the models. They hallucinate, run slowly, and cost more than expected. Sometimes, that's the real problem. But in the agent systems we've built, most of the engineering work went into solving the challenges mentioned above, not writing prompts.
A demo proves the model can do the task
That’s worth proving, and increasingly, the answer is yes. But in production, teams need to know more: how much the agent costs, who it acts for, how it fails, what it can access, and what data it keeps. A demo won’t answer these questions because it runs only once, for one user, under ideal conditions.
Cost grows with every step, not every task
A chat exchange is one request and one response. An agent task is a loop, and every turn of that loop sends the whole conversation so far back to the model. In data from Anthropic's own research system, agents used about 4 times the tokens of a chat interaction, and multi-agent systems about 15 times. The more steps a task takes, the faster its cost grows. A task with twice as many steps can cost more than twice as much. Caching helps lower the cost, but it doesn’t remove the problem.
In the same McKinsey survey, one in five respondents said AI costs, including token usage, already limit how much they can use AI.
A cheaper model cuts the bill by its price difference and no more, and a weaker model often needs more steps to finish the same task. The number of calls and the size of each one are the parts you control.
So, we control costs while the agent is running. We give every run a budget, with a warning before it's reached. The hard part is what happens at the limit. If the run simply dies, you've paid for half a task and got nothing for it. The run has to stop at a clean point with its state saved, so that raising the budget continues it rather than starting it again. That needs checkpointing, which is real work up front. We think it's worth it, because the alternatives are no limit at all or a limit that throws money away.
Prompt caching only pays off if each request is the previous request plus new messages. We learned how strict that is by measuring it: a single block that changed on every call, placed after the conversation, kept one provider's cache stuck at the system prompt for an entire run. Every call after the first paid full price for everything after the system prompted.
Every action needs a name attached
In a demo, the agent runs with a developer's API keys and can see everything. In production it acts for one user, inside one customer's data, and it may hand part of the job to a sub-agent.
When that sub-agent deletes a record, the audit log has to show more than "the AI did it". It needs the chain: who asked, which agent acted, and which agent it passed the work to. Identity was already a major security challenge before AI agents came along. According to Palo Alto Networks’ 2026 Identity Security Landscape, a survey of more than 2,900 security leaders, machine identities outnumber human identities 109 to 1. AI agents add to that number, with every agent and sub-agent creating another identity to manage.
The trade-off is friction. In the agent systems we build, every run has to name the customer it belongs to and the identity it acts for. There is no default and no anonymous mode, so tests have to set up identity the same way production does. We accept that. An agent that runs without an identity in testing will eventually run without one in production.
Long-running work has to survive a restart
Take the refund. A customer asks for one, and the agent checks the order, decides the refund is valid, and asks a manager to approve it. The manager is in another time zone and approves it the next morning. Overnight, the service is redeployed.
There are two possible outcomes. The agent either loses track of the task, leaving the approval request stuck, or starts over, checks the order again, and asks for approval a second time. When emails or payments are involved, doing something twice can become a serious incident.
Agents need what workflow engines have always had: state written down after each step, and a way to resume from the last safe point. Resuming isn't enough on its own, though. The refund call itself should carry an idempotency key, so that if the agent does repeat a step, the payment system recognizes it and refuses to pay twice.
They also need a way to stop, and that includes every agent after the first one starts, even those running on another server.
Regulation points the same way. Article 14 of the EU AI Act requires that the people overseeing a high-risk system can, "as appropriate and proportionate", interrupt it through "a 'stop' button or a similar procedure that allows the system to come to a halt in a safe state". A safe halt is the same thing a clean resume needs. The Digital Omnibus, in force since 27 July 2026, moved the deadline for stand-alone high-risk systems to 2 December 2027, and for systems built into regulated products to 2 August 2028. The obligation was deferred, not removed. (This isn't legal advice; check with your own counsel whether your system counts as high-risk.)
Tools turn prompt injection into code execution
Give an agent a tool and every piece of text it reads can try to use that tool. This year it showed up in the wild. In April, Google and Forcepoint each published what they had found on the open web: instructions hidden in ordinary pages, aimed at AI agents. Google described most of it as unsophisticated, and said attackers had not yet turned this kind of attack into an operation at scale. Forcepoint’s tests also included a hidden $5,000 PayPal transfer for agents that can make payments and a sudo rm -rf command for agents with access to the system shell.
In May, Microsoft's security researchers published an analysis of two flaws in its Semantic Kernel agent framework. In one, a value the model controlled reached Python's eval () through a blocklist that could be bypassed. In the other, a file-download function had been exposed to the model as a tool. Combined with code the agent wrote inside its sandbox, a prompt could place a script in the host's Windows Startup folder.
Microsoft’s recommendation is clear: treat anything the AI model can control as potentially unsafe input. We take an even stricter approach to shell access. By default, agents can’t run shell commands unless they’ve been explicitly given permission. Running commands without restrictions requires users to opt in, and the strictest setting blocks any command that can’t run in an OS sandbox.
That has a cost. Out of the box, an agent can't touch the shell on any operating system until someone grants it access, and developers used to the demo working will push back. The strictest setting is harsher still: there's no Windows sandbox we can rely on, so under it those commands are refused on Windows hosts. We'd still take that over the opposite default. Once people depend on permission, taking it away is an issue. Not granting it is just a default.
Memory needs an owner
Memory makes an agent feel smart across sessions, and it's the easiest way to show one person something that belongs to someone else. What matters most isn’t whether the agent remembers, but whose memory it saves information to and who can access it later.
Our default is narrow: an agent recalls only from the conversation it's in, unless it's configured to reach wider. A narrow default means some useful recall is missed until someone sets it up. If memory access is too broad by default, one configuration mistake could expose one user’s history to another user at the same company. From the outside, this can look just like the model making things up.
Traditional Controls for a New Era of AI Agents
Engineers have solved budgets, identity, crash recovery, sandboxing, and data isolation many times. What's new with agents is that the model chooses its own next step. You can't wrap these controls around a fixed sequence of calls, because there isn't one. They need to be in place wherever the model or its tools are called.
Before an ai-pilot starts, these five questions are worth answering in writing:
- What's the most one run may cost, and what happens to the run when it gets there?
- Whose identity does every action carry, including actions taken by sub-agents?
- If the process dies halfway, where does the work resume, which side effects are protected from repeating, and how do we stop it everywhere?
- Which tool arguments can the model influence, and what happens when it reads something hostile?
- Whose memory does the agent write to, and who can read it back?
Vague answers usually mean the pilot will stay a pilot. Our guide to building an AI strategy roadmap covers where agents fit in the business plan. These five questions cover engineering.
