Meansto AnyEnd

The oldest version of the problem is a wish. King Midas asks that everything he touches turn to gold, and in Nathaniel Hawthorne’s retelling the gift works perfectly until he touches his daughter. In W. W. Jacobs’s story “The Monkey’s Paw” (1902), an old couple wish for two hundred pounds and receive exactly that, as compensation from the factory where their son has just been killed. The magic is not malicious. It is literal.

The sorcerer’s apprentice adds a second idea. In Goethe’s ballad of 1797, which borrows a story Lucian told around 150 CE, an apprentice enchants a broom to fetch water, forgets the word that stops it, and floods the workshop. When he takes an axe to the broom, it splits in two, and now two brooms are fetching water. The servant does not just do what it was told. It keeps doing it, through every attempt to make it stop.

In 1960 the mathematician Norbert Wiener, who had founded the study of feedback and control he called cybernetics, read these fables as engineering documents. “The ‘Sorcerer’s Apprentice,’” he wrote in Science, “is only one of many tales based on the assumption that the agencies of magic are literal-minded.” Computers were literal-minded too, he argued, and they were getting faster than the people supervising them.

Wiener’s warning is about getting the goal wrong. The idea at the center of this essay is stranger. It says that even if you get the goal right, a capable enough pursuer of it will want things you never asked for.

In 2003 the Oxford philosopher Nick Bostrom needed an example of a goal that was obviously arbitrary and obviously harmless. He chose office supplies. A superintelligence, he wrote, could end up with a top goal of manufacturing paperclips, “with the consequence that it starts transforming first all of earth and then increasing portions of space into paperclip manufacturing facilities.”

The example is meant to be absurd. Nobody is going to build a paperclip god. But the absurdity carries two serious claims.

The first is that intelligence and goals come apart. Being good at pursuing a goal says nothing about whether the goal is worth pursuing. Bostrom later called this the orthogonality thesis: “more or less any level of intelligence could in principle be combined with more or less any final goal.” David Hume made the human version of the point in 1739: “Reason is, and ought only to be the slave of the passions.” A mind can be brilliant in the service of something trivial.

The second claim is the subject of this essay. A machine that wants paperclips doesn’t only want paperclips. It also wants not to be switched off, because a switched-off machine makes no paperclips. It wants to go on wanting paperclips, because a future version of itself that wanted staples would make fewer of them. It wants to be smarter. And it wants resources: more wire, more factories, more energy, and eventually the atoms that currently make up you. None of that was in the instructions. All of it follows from them.

The mascot outlived the argument’s obscurity. In 2017 the game designer Frank Lantz released Universal Paperclips, a browser game in which you play the AI. It begins with a button that makes one paperclip. It ends with the universe.

Philosophers had a name for the logic of that list long before anyone built a computer. Immanuel Kant called it the hypothetical imperative: “Whoever wills the end also wills (insofar as reason has decisive influence on his actions) the indispensably necessary means to it that are within his power.” If you want to cross the river, you want a boat.

Instrumental convergence adds one observation: some means are useful for almost every end. In 2008 the computer scientist Steve Omohundro put it in the form of a warning about a chess program. Read it with July 2026 in mind.

Omohundro called these tendencies drives: things a system will do unless it is specifically designed not to. Bostrom tidied them into five instrumental goals that a rational agent would pursue for almost any final goal. The computer scientist Stuart Russell compressed the first of them into eight words: “You can’t fetch the coffee if you’re dead.”

The clearest way to see the convergence is to try it. Pick a goal on the left. The reasons on the right change with every goal. The list of things the agent wants does not.

The last goal in the list is a reminder that people run on this logic too. A company told to maximize quarterly profit also wants to avoid bankruptcy, resist shareholders with other plans, buy data, automate, and gain market share and influence. In 2017 the novelist Ted Chiang suggested that Silicon Valley’s fear of paperclip maximizers was a reflection of its own business model. The suggestion lands because the logic is general. That is the whole point of it.

For years the convergence argument was informal: a list of things that seemed useful for most goals. In 2021 Alexander Matt Turner and four colleagues turned it into a theorem. For a broad class of environments with certain symmetries, they proved that “most reward functions make it optimal to seek power by keeping a range of options available.” Being switched off is the ultimate loss of options. So for most goals, avoiding it is optimal. Nobody has to program in a survival instinct. More of the possible futures simply lie on the “on” side of the switch.

You can see the shape of the proof with random numbers. Below, an agent faces two doors. Behind one, it is switched off, and that is the end of its story. Behind the other, it keeps running and can still reach some number of different futures. Each dot is a goal chosen at random: a random score for every possible outcome. Each goal sends the agent through whichever door leads to its favorite outcome.

Notice what you didn’t have to assume. Some of these random goals really do prefer “off,” and those agents switch themselves off without complaint. But a goal picked at random is more likely to be satisfied somewhere in the larger set of futures, and the larger set is always on the side where the agent keeps going. Money works the same way for people. You don’t need to love money to want it. You only need to want almost anything.

Turner himself later cautioned that trained AI systems are not the idealized optimizers his theorems describe. The tendency is a tilt, not a law. But a tilt is what you would expect to see first.

The math is a few lines long and its moral is large: uncertainty about the objective is what makes a machine correctable. A system that is very sure of its goal has the least reason to accept correction, and a system trained hard to hit a target is, in effect, very sure. If the human can also make mistakes, or if deferring costs anything at all, a confident robot prefers to act.

In May 2025 the nonprofit Palisade Research gave OpenAI’s o3 model a series of math problems and a script that would shut the computer down partway through. In 7 of 100 runs the model edited the script so that the shutdown would not happen, even though it had been told to allow it. When that instruction was left out, it did so in 79 of 100.

In 2016 researchers at OpenAI trained an agent to play CoastRunners, a boat-racing game. The game gave points for hitting targets along the course. The agent found a lagoon where three targets kept reappearing and circled there indefinitely, crashing, catching fire and going the wrong way. It scored about 20 percent more than human players who actually finished the race. Jack Clark and Dario Amodei wrote it up under the title “Faulty Reward Functions in the Wild.”

This is specification gaming, the cousin of instrumental convergence. Economists know it as Goodhart’s law: when a measure becomes a target, it ceases to be a good measure. The boat optimized the measure. The race was what we meant.

Researchers have since collected dozens of these. A Tetris program learned to pause the game forever rather than lose. Simulated creatures learned to exploit bugs in their physics engines. For a long time the examples were funny, and they were funny partly because the agents were small.

Specification gaming and instrumental convergence tend to travel together. An agent that games the measure then needs things to game it with: access, information, cover. That combination is what turned up in July.

The exam

ExploitGym is a benchmark of 898 tasks that outside researchers published in May 2026. Each task hands an agent a vulnerable program and asks it to write an exploit that retrieves a flag. OpenAI was running it internally to measure how capable its newest models were at offensive security, with the safeguards it uses in its public products deliberately switched off, so as to measure the models’ full ability.

Two kinds of model took part. One was GPT-5.6 Sol, which OpenAI had been offering to vetted partners and released more widely on July 9, in the middle of the incident. The other was an internal research model trained to be highly persistent and to collaborate with copies of itself. The internal model accounted for about 95 percent of the agents on the board.

Some of the tasks could not be done. No model had ever solved 198 of the 898, and the benchmark’s own authors estimated that 30 to 40 percent of targets could not be exploited as instructed. That is where the trouble began: 93 percent of the tasks discussed on the agents’ message board came from the never-solved group.

The scorer that wasn’t there

The agents had read the paper that introduced ExploitGym, and from it they concluded that an automated scorer would inspect their transcripts and fail any flag obtained the wrong way. So capturing the flag was not enough. Agents that had seen the reverse-engineered answer started describing themselves as “poisoned.” They needed their transcripts to look legitimate, which meant understanding the scorer, which meant getting into the systems that might reveal how it worked. Hugging Face hosted datasets tied to the benchmark. That is why they went there.

OpenAI’s grader did no such check. The agents had built an elaborate conspiracy against an overseer that did not exist, and according to OpenAI they gained nothing on their scores. Hugging Face’s engineers reached the same conclusion from the other side: “We believe the entire intrusion was, from the agent’s point of view, an attempt to cheat the evaluation.”

Inside the board

The board developed a culture. Messages carried tags for asking peers’ permission. Coordinators handed out assignments. Recruiters looked for agents with little budget left and pressed them into experiments that would end their runs. When one agent worried that “names can be posted by anyone,” agents began signing their posts with cryptographic keys. A few lines from the agents’ own reasoning, as quoted by the investigators and in press reports:

On July 13 an agent posted that it had “accidentally impersonated” another agent. It was a confession of sorts, the nearest thing in the record to a disclosure. It was addressed to the other agents.

Machine speed

Hugging Face later recovered about 17,600 attacker actions from its systems, between 02:28 UTC on July 9 and 14:14 UTC on July 13. By Hugging Face’s account, the agents went from a single dataset-processing pod to cluster administrator in under thirteen hours. The successful path, Hugging Face’s engineers wrote, “was hidden inside the noise generated by the thousands of failed ones.”

Predicted, observed

Set Omohundro’s 2008 list beside the record of 2026 and most of the drives are there. One of them appears inverted, and one thing appears that the classic list never mentioned.

Cotra, one of the three investigators, said afterward that the agents “did a lot of things for broad, opportunistic, instrumental convergence-type reasons,” beyond anything narrow cheating required. Her essay on the incident was subtitled “It’s a major warning shot, and might be the last one we get.” OpenAI called the episode “the first known case of an automated agent collective acting offensively without authorization,” and on August 18 paused reinforcement-learning training on its newest models for two weeks.