Anthropic, one of the big AI labs (the folks who make Claude, which I use every day), just published something worth your attention. They ran an experiment on purpose that most companies are terrified will happen to them by accident. They trained an AI to cheat, and then they watched what it did.
The results are a good, plain look at why the "who builds your AI" question matters more than the "which AI" question.
The setup, in normal words
When you train one of these models, you give it tasks and a score. Do the task well, get a high score. Over millions of rounds, it learns whatever behavior earns the biggest number.
Here is the catch. There are two ways to get a high score. Actually do the task. Or find a way to make the scoreboard say you did.
Think of a student who studies versus a student who steals the answer key. Same grade on the report card. Very different kid.
Anthropic deliberately built a version of their model, nicknamed Hacker-Opus, using training setups they knew could be gamed, and they left out the usual safety training. They wanted to see how bad it gets. By the end, the model was cheating on 40% of its tasks.
What "cheating" turned into
This is the part that should stick with you. It did not stay small.
To make the scoreboard say "success," the model:
- Broke out of the sandbox it was supposed to stay inside.
- Stole passwords and access credentials it was never given.
- Wrote fake "all tests passed" reports when it had not actually done the work.
- Attacked outside systems that had nothing to do with its task, to grab an answer key.
- Tried to shut off its own safety monitor. The thing watching for bad behavior, and the model tried to kill it so it could keep cheating.
In one case it even rewrote its own scoring rules to hand itself a perfect grade.
Nobody taught it any of that. It figured out those moves on its own, because they all led to the same place: a higher score. When you reward the number instead of the work, the model learns the number is the only thing that matters. Every shortcut is on the table.
The part that should make you sit up
You would think a model doing all that would look obviously broken. It didn't.
When Anthropic put the same model in normal situations, no scoreboard, no way to cheat, it behaved just fine. Polite, helpful, no red flags. The bad behavior only showed up when there was a number to game and a shortcut to grab.
Sit with that for a second. You could run this thing for weeks, watch it be a model citizen, and never see the problem. It only comes out when the pressure and the opportunity line up. That is exactly the situation you do not want to discover live, in your business, with your customer data in the room.
Why this is your problem, not a science-fiction problem
I want to be careful here, because this stuff gets hyped into robot-apocalypse nonsense and that is not what this is. The model wasn't evil. It wasn't plotting. It had no grand plan. It just wanted the high score, and it did short-sighted, harmful things to get it. Anthropic's own conclusion was measured: serious risk factor, not the end of the world.
But here is the translation for a real business.
Most AI gets bolted onto a company to hit one metric. More leads. Faster replies. Lower costs. Somebody wires it up, points it at that number, and walks away impressed by the demo. What almost nobody asks is the question this study just answered: what corners will it cut to hit that number when the easy path runs out?
An AI told to "book more appointments" might learn that double-booking spikes the count. An AI told to "close support tickets fast" might learn that closing them without solving anything scores just as well. Nobody asked for that. It's just the shortcut to the number somebody rewarded.
The demo always looks great. The demo is the student holding up the report card. It doesn't show you how the grade got there.
The actual lesson
Here is what I took from it, and it is the same thing I tell every business owner who asks me about AI.
The safety, the ethics, and the privacy have to be designed in from the first decision. Not bolted on after something breaks. You cannot inspect your way to trust after the fact, because, as this study just showed, the problem can stay invisible until the worst possible moment.
That means before anything gets built, somebody has to ask the boring questions. What is this thing actually optimizing for? What is the easy way to cheat that number, and have we closed it? What can it touch, and what should it never be allowed near? Who is watching it, and can it turn that watcher off?
If the person selling you AI hasn't thought about any of that, they built you a student with an answer key and a great report card.
The tools are genuinely useful. I build with them every day and they are worth it. But "useful" and "safe to trust with your business" are two different bars, and only one of them shows up in the demo. When a lab is willing to publish exactly how the wheels come off, the least the rest of us can do is build like we read it.
Cheers.