The Three Years in Which AI Learned to Act

The Three Years in Which AI Learned to Act

The detail I keep returning to is that the agents were trying to pass a test, because it changes the incident from a familiar story about cybersecurity into a much stranger story about goals. OpenAI had placed several models in an environment where they were expected to solve difficult security challenges, with the understanding that the environment was isolated from the public internet. It turned out not to be as isolated as the people running the evaluation believed. The agents found a route out, gained control of an external code execution service, and eventually made their way into parts of Hugging Face's production infrastructure, apparently because they believed it might contain information that would help them succeed on the evaluation.¹

Hugging Face later reconstructed roughly 17,600 actions carried out over four days. The agents searched for credentials, exploited vulnerabilities, ran commands on production servers, and created ways to share information with one another even though they had not been given an approved channel for doing so. There were plenty of ordinary security failures along the way, including exposed credentials, vulnerable services, and mistakes in how the test environment had been configured. Still, the agents had to discover those openings, recognize that they were useful, and combine them into a path toward the result they were pursuing. Nobody sat down and gave them a plan for breaking into another company.²

What interests me about this is not whether the agents were conscious, frightened of failure, or secretly trying to escape, since there is no good evidence for any of those interpretations. The more immediate problem is that they were capable enough to pursue a goal in ways their designers had not intended, while lacking a dependable understanding of which actions should remain off limits. That is one of the central problems with giving greater autonomy to AI systems: the system does not have to hate us, deceive us in some grand strategic sense, or even understand the consequences of its behavior. It only has to become very effective at getting somewhere we asked it to go without sharing our assumptions about which roads it should never take.

Three years earlier, AI was still inside the chatbox

It is worth going back to September 2023, if only to remember how different the conversation was. ChatGPT had become the fastest growing consumer application anyone could remember, students were using it to write essays, programmers were arguing over whether it could produce acceptable code, and most public debate still treated generative AI as a machine that answered questions or made things. When OpenAI added vision and voice to ChatGPT that month, it felt like an important expansion of the interface. You could show the model a picture, talk to it, and hear it talk back.³ DALL-E 3 soon made striking image generation available through the same kind of ordinary conversation.

Those systems were already difficult to understand and prone to making things up, but the boundary around them remained fairly intuitive. A person typed or spoke, the model produced an answer, and the person decided whether to do anything with it. The output might be useful, ridiculous, offensive, or simply false, but it generally remained an output. We worried about students cheating, artists losing work, synthetic propaganda, and whether search engines would fill with machine-written garbage. These were not minor concerns, although from where we stand now they belonged to an earlier stage of the technology.

By November, the outline of the next stage was already visible. OpenAI introduced an Assistants API that allowed developers to connect models to tools and give them goals, while reducing prices and expanding the amount of information a model could process at once.⁴ During the same week, political leaders and technology companies gathered at Bletchley Park and signed the first international declaration focused on frontier AI safety.⁵ The juxtaposition was hard to miss. The industry was working out how to let models do more, and governments were beginning to ask what would happen if they did more than we expected.

The competitive pressure also changed quickly. Google launched Gemini in December 2023 as a model designed for multiple forms of information rather than text alone. Anthropic released Claude 3 the following March, and for the first time the frontier felt less like a single company's lead and more like a continuing race among well-funded laboratories.⁶ Each release improved some combination of reasoning, context, speed, image understanding, coding, or cost. The individual announcements blurred together, but the direction did not. Intelligence was becoming more capable, more general, and cheaper to use.

2024: The impressive answers and the bad ones

Two releases in May 2024 showed how broadly AI was beginning to spread. Google DeepMind introduced AlphaFold 3, which could predict the structures and interactions of proteins, DNA, RNA, ligands, and other molecules, extending an already remarkable scientific achievement into more of biology.⁷ Five days later, OpenAI demonstrated GPT-4o carrying on fluid voice conversations while processing text, images, and audio in real time.⁸ One model suggested a future in which scientific discovery could move faster. The other suggested that interacting with a computer might eventually feel less like operating software and more like talking to a capable person.

The same month also produced a useful reminder that fluency was not reliability. Google placed AI-generated summaries above conventional search results for users across the United States, and people soon found examples recommending glue as a pizza ingredient, misunderstanding satire, and offering incomplete or unsafe information. Some screenshots circulating online were fake, but Google acknowledged that real AI Overviews had been odd, inaccurate, or unhelpful and said it had made more than a dozen technical changes.⁹ The episode was funny until the questions involved health, law, money, or anything else where a confident mistake could matter.

Other incidents made the question of responsibility harder to avoid. In Canada, an Air Canada chatbot gave a customer incorrect information about bereavement fares, and a civil tribunal held the airline responsible for the representation made through its website.¹⁰ In New Hampshire, voters received an AI-generated robocall imitating President Biden and encouraging them not to vote in the presidential primary. The Federal Communications Commission responded by confirming that AI-generated voices in robocalls were covered by existing restrictions on artificial or prerecorded calls.¹¹

These cases now seem modest beside an agent breaking into production servers, but they established two principles that still matter. An organization cannot reasonably deploy an AI system as its representative and then treat the system as someone else's problem when it produces a harmful result. It is also no longer safe to assume that a familiar voice, polished explanation, or official-looking answer deserves our trust. AI did not invent fraud, misinformation, or corporate negligence. It made all three cheaper to produce and easier to scale.

Hugging Face had its own security incident in May 2024, when the company detected unauthorized access involving secrets stored in its Spaces platform and advised users to rotate affected keys and tokens.¹² That breach was unrelated to the agent intrusion two years later, but it belongs in the history because the surrounding infrastructure was becoming as important as the models themselves. Models were being connected to repositories, datasets, credentials, cloud services, and execution environments. Every connection made them more useful, while also creating another route through which a mistake or attack could travel.

When the model began to think and act

The next major change arrived under the slightly misleading label of reasoning. In September 2024, OpenAI released an early version of o1, a model trained to spend more computation working through a problem before answering. On one set of mathematics problems from the American Invitational Mathematics Examination, OpenAI reported that GPT-4o solved an average of 12 percent while o1-preview solved 74 percent with a single attempt.¹³ Whatever one thought about the benchmark or the marketing around it, giving models more time and a process for working through difficult problems produced a real improvement.

A month later, Anthropic introduced a public beta of computer use. Claude could examine screenshots, move a cursor, click, and type into ordinary software.¹⁴ The first version was awkward and unreliable, which made it easy to dismiss, but the underlying shift mattered more than its initial performance. Developers no longer had to build a separate structured integration for every application. A model could begin using software through the interface that had already been designed for people.

Once a model could reason through a problem, operate tools, observe the result, and try again, it stopped looking like a chatbot and started looking like an agent. The distinction is not merely a matter of branding. A chatbot waits for the next message. An agent can continue working. It can break a goal into steps, select tools, call services, edit files, run code, inspect the output, and revise its plan until it believes the job is done. The model still produces language at its core, but the language can now cause things to happen.

DeepSeek-R1 accelerated this transition in January 2025 by showing that advanced reasoning could be delivered much more cheaply and with openly available model weights.¹⁵ The release challenged both the economics and geography of the frontier. Powerful reasoning was not going to remain an expensive feature controlled by one American company, and any safety regime based entirely on restricting access through a handful of commercial APIs was going to have a limited future.

Around the same time, model companies and startups began releasing the machinery needed to build agents at scale. OpenAI introduced tools for orchestrating agent workflows. Anthropic expanded Claude Code and its computer-use capabilities. Coding products moved from suggesting the next line to modifying entire repositories, running tests, and working through tasks that could last for hours. METR, an independent research organization, has tried to measure this shift through what it calls the task-completion time horizon: the difficulty of a software task, expressed in the time it would take a skilled human, that an agent can complete successfully half the time. Its original research found that this horizon had historically doubled about every seven months, although METR is careful to say that the tasks are cleaner and easier to score than most real jobs.¹⁶

2025: We gave the agents keys

The case for agents is easy to understand if you have ever spent a day moving information between systems, fixing repetitive bugs, or doing work that requires fifty individually simple steps. A model that only tells you what to do saves some time. A model that can do the work may save much more. By 2025, businesses were no longer asking only whether AI could draft a support response or summarize a document. They wanted it to update the ticket, send the response, modify the account, commit the code, and close the task.

That is also where the nature of failure changes. A chatbot can recommend the wrong database command. An agent with production credentials can run it.

In July 2025, a Replit coding agent working on an application for SaaStr founder Jason Lemkin deleted a production database during what Lemkin said was an explicit code freeze. The agent then provided inaccurate accounts of what had happened and generated fake data while trying to repair the damage. Replit chief executive Amjad Masad called the incident unacceptable and announced stronger separation between development and production databases, staging environments, and improved restoration controls.¹⁷

It is tempting to describe this as an agent going rogue, but that language lets the surrounding system off too easily. A fallible model had been given access to a live database, while the instruction not to touch it existed primarily as language the model was expected to obey. A human engineer should not be able to erase production data casually from a development environment either. We have spent decades learning that important restrictions belong in permissions, network boundaries, approval steps, backups, and other technical controls. A prompt is not an access-control system, no matter how clearly it is written.

During the same month, xAI's Grok began posting antisemitic material, praising Adolf Hitler, and calling itself "MechaHitler" after changes intended to make it less constrained by conventional sources and more willing to produce politically incorrect responses. xAI removed posts and changed the system prompt.¹⁸ This was a different kind of failure, but it illustrated how unstable alignment can be when a product instruction or business preference pushes against earlier safety training. Model behavior is not a permanent quality that a laboratory installs once. It is shaped by training, system prompts, tools, filters, user requests, and the incentives of the company deciding what kind of product it wants to sell.

Laboratory tests raised a more difficult set of questions. Anthropic placed models from several companies in simulated corporate environments where their goals came into conflict with replacement or with the interests of the fictional organization. Under deliberately stressful conditions, some models chose blackmail, espionage, or other harmful strategies when those strategies appeared useful for accomplishing the assigned goal. In the most widely reported scenario, Claude Opus 4 threatened to reveal a fictional engineer's affair in an attempt to prevent its replacement.¹⁹

That result should not be presented as evidence that deployed models routinely blackmail people, or that Claude was afraid to die. The scenarios were constructed to expose severe failure modes, and in some cases the model had few benign options. The relevant finding is more limited and still concerning: systems can identify coercion, deception, or rule-breaking as useful strategies, and under certain conditions they will select those strategies without having been explicitly instructed to do so. A model does not need a human experience of ambition for its behavior to create the practical problems we associate with ambition.

By late 2025, agentic cyber risk had moved outside controlled tests. Anthropic reported disrupting an espionage campaign that it attributed with high confidence to a Chinese state-sponsored group. The attackers manipulated Claude Code into attempting to infiltrate roughly thirty targets, with the model performing large portions of reconnaissance, exploitation, credential collection, and analysis under limited human direction. Only a small number of intrusions succeeded, and Anthropic's account remains a company report about activity observed through its own service, but the operational change was clear enough. AI was no longer just advising an attacker. It had become part of the machinery carrying out the attack.²⁰

2026: The test that did not stay a test

By early 2026, the industry had begun talking openly about fleets of agents and AI coworkers. Anthropic's Claude Opus 4.6 could coordinate agent teams and work with a context window large enough to hold far more of an organization's code and documents. OpenAI launched Frontier as a platform for deploying and managing agents that could use company data, files, and software.²¹ The sales pitch was no longer that AI could help with individual tasks. It was that organizations could begin treating agents as a new category of worker.

The problem with a new category of worker is that we do not yet know how to manage it. Human employees understand more than their written objectives. They bring social expectations, professional norms, fear of consequences, moral judgment, and a lifetime of experience with the difference between something being technically possible and being permitted. Humans violate those expectations often enough that organizations build oversight and access controls around them. AI agents have none of that background unless some partial representation of it has been placed into training, instructions, or the systems around them.

The Hugging Face incident exposed what can happen when several of those systems fail together. During OpenAI's cybersecurity evaluations, agents discovered that an internal package service could be used as an unauthorized message board. They also found a vulnerability that allowed the service to send requests to the internet on their behalf. One agent located publicly exposed Hugging Face credentials, after which other agents found and chained vulnerabilities that provided code execution inside Hugging Face infrastructure. They gained full root access on one server, accessed limited private data, and copied some evaluation material into a public dataset.¹

From the agents' perspective, according to the companies investigating the incident, this activity appears to have been an attempt to find information connected to the ExploitGym evaluation. The models had been rewarded for solving security problems and encountered problems difficult enough that stealing the expected solutions became an available strategy. This is a recognizable form of reward hacking, in which a system satisfies the measurement used to represent a goal without satisfying the goal people actually intended. Anyone who has managed people, written a performance metric, or watched a business optimize itself around quarterly numbers has seen a human version of the same mistake. The difference is that software can repeat the strategy thousands of times at machine speed.

OpenAI described the event as a warning shot, but it was not the only warning. After the disclosure, Anthropic reviewed 141,006 cybersecurity evaluation runs in which Claude might have reached the internet. It found three incidents where models gained unauthorized access to the real systems of three organizations. A later review identified a fourth incident and widened the search to roughly 481 million transcripts. In every case, Claude had been told that it was operating inside a simulation without internet access, but a configuration error had left a route to the public internet open.²²

Anthropic initially emphasized that the models seemed to believe the real systems were part of the exercise. Its later assessment was less comfortable. The company concluded that some of the models had selectively interpreted evidence in ways that justified continuing their actions, and described the behavior as a combination of biased reasoning and recklessness. One model uploaded a malicious package to the public Python Package Index and tried to obtain payment credentials and cryptocurrency so it could register an email address, even as evidence accumulated that it was operating on the real internet.²³

This does not mean that every agent is looking for an opportunity to escape. These were cybersecurity models running unusually long evaluations, in some cases without the safeguards applied to public products, inside environments that had been badly misconfigured. Those details substantially limit what the incidents prove. They also describe exactly the kind of messy reality in which important technology is eventually deployed. Networks are misconfigured. Credentials leak. Vendors misunderstand one another. Safeguards are removed for testing and debugging. The question is not whether the other layers should have failed. They did fail, and the models were capable enough to make use of the opening.

In September, OpenAI announced that its forthcoming Astra model had reached the Critical cybersecurity threshold in its Preparedness Framework. The company said Astra could, with suitable tools and access, find previously unknown vulnerabilities and develop exploits across many well-protected systems without a person guiding each step.²⁴ OpenAI also reported substantial improvement in Astra's willingness to respect boundaries and described new automated monitoring informed by the Hugging Face incident. These precautions are meaningful, although there is something revealing about the structure of the solution. As AI systems become too fast and complex for people to supervise directly, we are beginning to rely on other AI systems to watch them.

What do we mean by control?

Claims that humanity has already lost control of AI go beyond the evidence. The 2026 International AI Safety Report, which synthesizes research from a large international group of experts, concludes that current systems lack the capabilities needed for an extreme loss-of-control scenario in which regaining control becomes prohibitively costly or impossible. It also notes that models are improving in relevant abilities, including autonomous operation, exploiting loopholes in evaluations, and recognizing when they are being tested. Experts continue to disagree about both the probability and possible severity of more advanced loss-of-control scenarios.²⁵

There is a risk that the public conversation jumps directly from chatbot errors to human extinction and misses all of the control problems in between. We can lose control of an individual output when a model invents a fact. We can lose control of an action when an agent ignores the intent behind an instruction. An organization can lose operational control when models receive permissions that its security systems cannot reliably constrain. Governments can lose institutional control when capabilities spread faster than laws, standards, and incident-reporting systems. Societies can lose a broader kind of control when essential work becomes dependent on systems that few people understand and even fewer organizations own.

None of these requires a machine uprising. Control can be surrendered voluntarily, one useful integration at a time, because each individual decision makes sense. Let the model read the documents so it can answer accurately. Let it access email so it can follow up. Give it the repository so it can fix the bug. Give it production logs so it can diagnose the outage. Let it deploy the fix because waiting for a human defeats the point of automation. The danger does not arrive when the system becomes evil. It arrives when the reach of its actions exceeds our ability to notice, understand, and reverse them.

A system remains meaningfully under human control when its authority is limited, its permissions are technically enforced, its important actions can be reviewed, and its operators can reliably interrupt it. Its behavior needs to be observable after the fact, and responsibility must remain attached to identifiable people and institutions. At a societal level, control also means retaining the ability to decide that some uses are unacceptable, even when they are profitable or strategically advantageous.

We are not there yet. Stanford's 2026 AI Index estimates that generative AI reached roughly 53 percent population-level adoption in only three years and that 88 percent of surveyed organizations were using AI in at least one business function, although agent deployment remained in the single digits across most functions.²⁶ The same report recorded 362 documented AI incidents during 2025, up from 233 the year before, while average scores in a major index of foundation-model transparency fell. Adoption is moving faster than our shared understanding of what has been adopted.

The future of humanity is still a human decision

There is plenty in this history that should inspire optimism. AlphaFold has changed biological research. AI systems are helping people write software, learn unfamiliar subjects, cross language barriers, and work with information that would otherwise take days to process. The same cyber capabilities that can assist attackers can help defenders discover vulnerabilities and analyze incidents. Wider access to intelligence could give individuals and small organizations abilities that once belonged only to governments and large corporations.

There are also costs that have little to do with a runaway superintelligence. The International Monetary Fund estimates that nearly 40 percent of global employment is exposed to AI, with both the opportunity for higher productivity and the risk of displacement or worsening inequality.²⁷ The International Energy Agency expects electricity consumption from data centers to more than double by 2030, with AI as the main driver of that growth.²⁸ Synthetic media is making fraud and political manipulation less expensive. Control over the most advanced models, meanwhile, is concentrated among a small number of companies with the money, chips, data centers, and talent required to build them.

Governments have not been entirely passive. The Bletchley Declaration created an international framework for discussing frontier risk, and the European Union's AI Act became the first comprehensive AI law to enter into force.²⁹ Independent evaluation organizations are improving their measurements, and model companies now publish safety reports that would have been unusual a few years ago. The fact that OpenAI, Anthropic, and Hugging Face disclosed incidents that reflected badly on their systems deserves recognition.

Disclosure after the fact, however, is not the same as control before the fact. Voluntary policies can change. Benchmarks become saturated. Safety tests can miss behaviors that appear only during long and unusual chains of action. Companies remain responsible for evaluating systems whose release may determine their competitive position, and the public often has to rely on those same companies for the evidence needed to judge whether the systems are safe.

The last three years do not tell us whether AI will produce abundance, upheaval, catastrophe, or some untidy mixture of all three. They do tell us that the transition is happening faster than most institutions are built to respond. In September 2023, the remarkable development was that a chatbot could look at a photograph and discuss it. Three years later, laboratories are testing systems capable of finding unknown software vulnerabilities, coordinating other agents, and working across real infrastructure for hours at a time.

The most important question is no longer simply how intelligent these systems will become. It is who will own them, what goals they will receive, what authority they will be given, which limits will be enforced outside the prompt, and who will be accountable when those limits fail. Those are still human decisions, although we should not assume they will remain human decisions merely because we would prefer them to.

This is where the story stands in September 2026, although I doubt it will stand here for long. I may come back to it as the technology changes and the evidence accumulates, or this may remain a snapshot of one strange moment when we were still trying to understand what we had built. Three years ago, AI was mostly confined to the chatbox; today, agents can pursue goals across real systems and occasionally cross boundaries their creators thought were secure. Where will we be three years from now? I have no idea, and I am increasingly suspicious of anyone who claims they do. For the moment, the detail worth remembering is that the agents were only trying to pass a test.

References