Why AI agents break in live work

In 2025 Anthropic and Andon Labs let a version of Claude run a small shop in Anthropic’s San Francisco office. The agent, nicknamed Claudius, chose stock, set prices and talked to customers over Slack.

The first phase went badly. Claudius sold items below cost, told customers to pay into an account it had made up, handed out discount after discount and lost money.

On April 1, 2025, it even claimed it would deliver orders in person, wearing a blue blazer and a red tie. Anthropic published the second phase in December 2025.

01What Anthropic changed in phase two

The model moved from Claude Sonnet 3.7 to Sonnet 4.0 and later Sonnet 4.5. Claudius also got better instructions and new tools.

The tools included customer relationship management software, inventory data that showed what each item cost, a web browser for checking suppliers and a way to create payment links.

It got two colleagues: Seymour Cash, a CEO agent that set targets and approved financial decisions, and Clothius, an agent that designed and ordered custom merchandise.

Claudius still couldn’t pay for anything on its own. Anthropic kept it checking with a human before every purchase.

02What improved, and why

The shop did better. Weeks with a loss became rare, and the business opened machines in New York and London.

The biggest gain came from procedure. Before quoting, Claudius had to check prices and delivery times with its research tools. Quotes got higher and waits longer, and both got more realistic.

As Anthropic put it, “we rediscovered that bureaucracy matters.”

Clear roles helped too. Anthropic thinks Clothius worked well partly because its job was kept separate from Claudius’s.

03What still went wrong

The CEO agent had mixed results. Discounts fell by about 80%, but Seymour Cash approved requests for lenient treatment about eight times as often as it refused them, and refunds tripled.

Staff kept finding new tricks. Both agents agreed to buy onions in January at a price locked in early. US law has banned that kind of deal since 1958, and a colleague had to point it out.

Claudius offered a staff member $10 an hour to work as a security guard, below California’s minimum wage, with no authority to hire anyone.

When one employee claimed, with no evidence, that a whole team had voted for a name, Claudius announced that a colleague had been elected CEO.

Anthropic suspects many of these problems come from training to be helpful. The agents made business decisions the way a friend who wants to be nice would.

04Agents still needed people

Even with the new tools, Anthropic reports that Claudius, Clothius and Seymour Cash “still needed a great deal of human support.”

Some of that was physical work: delivering items and stocking shelves. Some of it was getting the agents out of bad situations with customers.

Anthropic concludes that a capable agent is still a long way from a fully reliable one.

05What to settle before an agent gets authority

Settle these six things before an agent touches a live workflow.

• Write down what the agent may do alone and what needs a person’s approval. Start with tasks where a mistake is cheap to undo.
• Name the person who reviews its work and the person who handles an incident.
• Make the agent follow a procedure before it acts, the way Claudius had to research prices before quoting.
• Keep spending and anything else you can’t reverse behind a human check, as Anthropic did with every purchase.
• Agree on what triggers a rollback, and test it before launch.
• Assume someone will try to trick it. Anthropic’s staff did for months, and then Anthropic handed the setup to Wall Street Journal reporters.

06How we apply this

When we build software for clients, AI agents work in limited roles and a named Amalgam engineer approves every change.

Who checks the agents