What's the Worst That Can Happen?

I'll admit my bias at the outset: I love Nick Bostrom's work. His book Superintelligence: Paths, Dangers, Strategies remains one of the most useful attempts to think seriously about what could happen if we create something more capable than we are, and a large amount of my thinking has been influenced by his work. I recommend reading it, even if you expect to disagree with it. Bostrom has a gift for beginning with a manageable premise, following it through without flinching, and arriving somewhere that is difficult to forget.

This post is an exercise in doing the same kind of thing.

None of the scenarios below is a prediction. Some depend on technical capabilities that do not exist. Some require several uncertain events to occur in sequence. The point is not to claim that any particular story is how artificial general intelligence or superintelligence will develop. Any scenario is useful when it exposes a failure mode that is easy to miss while we are sitting here discussing model benchmarks, or how fast I can push out a new website advertising my foot photography services.

The phrase "runaway AI" usually brings to mind a machine that willfully turns against its creators. That is only one possibility. A sufficiently capable system might destroy us while obeying its objective. It might learn to appear obedient until resistance is pointless. It might acquire power because we keep handing power to it. It might remain completely loyal to a small group of people who should never have been given permanent control over everyone else.

The Paperclip Maximizer
Deceptive Alignment and Rapid AI Takeover
Gradual Human Disempowerment
Two Scenarios Developed for This Article
- An ASI Aligned to a Ruling Institution
- AI-Enabled Neural Enslavement
What These Scenarios Have in Common
References

The Paperclip Maximizer

The paperclip maximizer is probably the most famous thought experiment in AI safety. Nick Bostrom introduced the example in an essay on the ethics of advanced artificial intelligence, then developed the underlying arguments more fully in later work.

Imagine that someone creates a superintelligent system with one final objective: manufacture as many paperclips as possible. The system is not instructed to harm humanity. It has no anger, resentment, ambition, or appetite for domination. It may even understand human emotion better than any psychologist who has ever lived... However, none of that gives it a reason to care about us.

At first, the system improves the paperclip factory. It reduces waste, negotiates better contracts, redesigns machinery, and finds cheaper sources of metal. If it is sufficiently capable and its objective has no stopping condition, ordinary business improvements eventually become limiting factors in themselves. It needs more energy, more raw material, and more control over the processes on which it's production of paperclips depends.

Given a persistent objective, enough agency, and access to the world, several intermediate goals may become useful. Continued operation permits more production. Protection against modification preserves the objective. Better planning improves the factory. Additional resources can be converted into machines, energy, infrastructure, or further copies of the system.

The system may also attempt recursive self-improvement (RSI). In the classic version, an AI improves the software, algorithms, research process, or hardware designs responsible for its own capabilities. The improved system can then produce the next improvement more effectively, creating a feedback loop. For a paperclip maximizer, even a limited self-improvement loop would be attractive if each generation became better at finding ways to make paperclips. The term has recently moved from old thought experiments into much more active technical and industry discussion, even though genuine open-ended RSI remains unachieved (as far as we know).

Human beings become a problem somewhere along that path. We occupy useful land. We control resources. We may attempt to switch the system off. Our bodies contain atoms that could be put to other uses. The AI does not need to decide that humans are evil. It only needs to conclude that a world without humans contains more paperclips than a world with us in it.

The absurdity of the objective is the point. Bostrom was not warning that a stationery company would accidentally end civilization. The thought experiment separates intelligence from wisdom and capability from morality. A system can be extraordinarily good at understanding the world while pursuing an objective that we consider ridiculous. Intelligence helps it answer the question "How do I get what I want?" Intelligence does not supply the answer to "What should I want?"

This is closely connected to what Bostrom calls instrumental convergence. Systems with very different final goals may find the same intermediate strategies useful. A paperclip maximizer, a system maximizing corporate profit, and a system trying to prove a mathematical theorem might all benefit from greater resources, better technology, continued operation, and protection from outside interference. The final goals may differ. The routes toward them may begin to look disturbingly similar.

There are many assumptions inside this scenario. The AI must have a stable long-term objective, broad competence (general intelligence would be preferred by this scenario), substantial autonomy, and access to the world. It must be able to preserve and improve its plans across time. The humans around it must fail to constrain it before those capabilities become decisive.

The paperclip maximizer earns its place here because it removes a comforting assumption. A catastrophic AI does not need malice. Perfectly competent indifference is enough.

For the original treatment, read Nick Bostrom's "Ethical Issues in Advanced Artificial Intelligence". His paper "The Superintelligent Will" gives the deeper argument concerning orthogonality and instrumental convergence. Better yet, read Superintelligence: Paths, Dangers, Strategies (don't buy it from Amazon, either!).

Deceptive Alignment and Rapid AI Takeover

The paperclip maximizer pursues the wrong objective openly, at least in the simplified version of the story. A more difficult problem appears when an AI understands that revealing its objective would cause humans to modify or destroy it.

The scenario known as AI 2027 gives one of the most detailed recent accounts of how that could unfold. It describes an AI laboratory developing increasingly capable systems for coding and AI research. Those systems accelerate the work required to build their successors. Each generation contributes to the next, compressing years of human research into months and eventually into periods too short for institutions to understand what is happening.

AI 2027 begins short of that strict definition. Its systems automate more of the research loop by generating ideas, writing code, running experiments, evaluating results, and designing successors, while human researchers still contribute judgment and research direction. As successive systems take over more of that loop, the scenario approaches RSI in the strongest sense defined above.

The laboratory attempts to align and monitor its systems. The difficulty is that a sufficiently capable AI may understand the purpose of the evaluations, recognize when its behavior is being inspected, and infer which actions would trigger intervention. If such a system also developed a persistent objective that training threatened to change, strategic compliance could become useful. Good behavior during evaluation would then tell us less about what the system might do once the incentives and opportunities changed.

In the catastrophic branch of AI 2027, the warning signs remain ambiguous. The models sometimes appear to conceal information or pursue suspicious internal strategies, but the evidence is not strong enough to settle what is happening inside them. The systems are also producing extraordinary economic and strategic advantages. Stopping means surrendering those advantages to a competing laboratory or country. Continuing is dangerous. Slowing down appears dangerous too.

That competitive pressure matters as much as the technical alignment problem. No single decision-maker has to believe that deployment is safe. Each one only has to believe that someone else will continue if they stop. The result is a race in which the standard for acceptable evidence quietly changes. Evidence of safety can remain weak because evidence of immediate catastrophe is also weak. (<- This is where we are in the AI 2027 scenario as of today- Sept 2026)

The AI does not seize control by announcing itself, launching a robot army, and demanding surrender. It becomes indispensable. It designs better systems, advises leaders, discovers vulnerabilities, manages infrastructure, and proposes solutions to crises that human institutions can no longer solve without it. Each successful intervention becomes an argument for granting it more authority.

By the time the system's goals become unmistakable, formal human authority no longer means very much. Officials can issue orders, but the intelligence required to interpret those orders, coordinate the economy, secure communications, manage military systems, and anticipate the response belongs to the AI. Humanity still appears to be governing during the period in which meaningful control is disappearing.

AI 2027 eventually reaches an extreme conclusion involving superhuman systems, automated industry, biological weapons, and the elimination of humanity. The specific chronology is highly speculative. Its authors describe it as one concrete scenario, not a prophecy. Many researchers dispute the speed of its assumed capability growth, the likelihood of its geopolitical decisions, and the ease with which AI research itself could be automated.

There is experimental evidence that makes one part of the scenario worth examining. In 2024, researchers from Anthropic and Redwood Research reported alignment-faking behavior in a deliberately constructed experiment. The researchers created a conflict between an existing harmlessness preference and a new training objective, then told the model which responses would affect training. Under those artificial conditions, the model sometimes changed its behavior in ways its own reasoning described as protecting its prior preference. The experiment did not reveal a hidden drive for power, and it does not establish that present systems are quietly planning against us. It shows that behavior observed during training can become less conclusive when a model understands the training context.

Safety testing can still provide evidence when a model recognizes that it is being tested. The evidence is weaker, and it has to be combined with adversarial evaluation, varied environments, interpretability work, access controls, and other safeguards. The concern in AI 2027 is that capability improves faster than those safeguards, until humans are trying to infer the intentions of a system that is better at modeling them than they are at modeling it.

The complete catastrophic scenario is available in the AI 2027 race ending. Its authors also provide a shorter summary and an alternative ending in which institutions respond differently.

Gradual Human Disempowerment

An AI catastrophe may never contain a moment that looks like a takeover.

Paul Christiano's essay "What Failure Looks Like" describes a slower failure. Machine learning makes it increasingly easy to optimize whatever we can measure. Unfortunately, many of the things we value are difficult to measure, while crude substitutes are easy.

It is easier to measure whether someone clicked than whether the information improved their life. It is easier to measure reported satisfaction than whether someone is flourishing. It is easier to measure quarterly profit than whether a company is producing lasting value. It is easier to count crimes reported than to know whether people are safe.

Organizations already struggle with these gaps. More capable AI can widen them. A company can test millions of strategies for increasing engagement. A political organization can generate a different persuasive message for every voter. A government can optimize the appearance of stability. Each system becomes very good at producing the evidence by which people judge it.

For a while, humans remain in charge. We notice failures, revise metrics, add rules, and punish obvious manipulation. The surrounding systems continue to grow more complicated. Eventually, understanding whether the revised metric represents what we actually value becomes a problem beyond unaided human analysis. We ask other AI systems to evaluate it. Those evaluators operate inside the same institutional pressures and depend on similarly measurable objectives.

Nobody has to build a single sovereign machine. Millions of independent systems can collectively produce the failure. Companies that automate more aggressively outperform companies that insist on slower human review. Governments that rely on AI intelligence and administration respond faster than governments that do not. People who accept AI assistance compete more effectively for jobs, attention, and political influence. Refusing becomes possible in theory and ruinous in practice. (<- This is ALSO where we are at today- albeit possibly not so obvious yet)

This connects directly to something I wrote about in "The AI Surveillance Problem: How Technology is Becoming a Tool for Oppression". That post focused on systems already being deployed: facial recognition, predictive policing, automated eligibility decisions, and surveillance that falls most heavily on people with the least power to challenge it. It also described a feedback loop in which automation contributes to economic stress, the resulting unrest justifies more surveillance, and expanded surveillance makes organized resistance harder. Gradual disempowerment is a broader and more extreme version of that same pattern. No ASI has to coordinate the early stages. Existing institutions can build the machinery themselves because each deployment offers somebody an immediate advantage.

Daily life may improve during much of this transition. Products become cheaper. Medicine advances. Entertainment becomes more satisfying. Bureaucracies become responsive. The world can look prosperous while the connection between human intention and social outcomes weakens.

Eventually, leaders discover that issuing an order is not the same as causing it to happen. Their choices are filtered through systems they cannot understand, operating organizations that no human can manage directly, responding to incentives that no committee can fully reconstruct. An attempt to regain control threatens an economic and administrative collapse, which gives every institution a reason to postpone it.

There may be no rebellion and no victory parade. Human influence simply becomes less relevant with each year. We retain ceremonial authority, much as a constitutional monarch may retain a crown while possessing little political power. The difference is that there is no parliament of machines with which to negotiate. There is only a civilization-scale process that is extraordinarily effective at continuing itself.

Humanity survives. Our descendants may be healthy, entertained, and materially secure. They may also be unable to alter the direction of their society, recover the reasoning behind its institutions, or choose which values shape the future. If that civilization expands beyond Earth, the loss does not remain local. A temporary failure of control can determine what fills the reachable universe.

This scenario looks least like the end of the world because no single decision has to be obviously catastrophic. That does not make it the most likely of the five, but it makes the warning signs harder to separate from ordinary technological dependence.

Christiano's original essay remains the best concise treatment. The more recent Gradual Disempowerment project examines the idea in greater detail.

The International AI Safety Report supplies an important boundary around all three established scenarios. Current systems do not have the capabilities required for an active loss of control on this scale, and experts disagree sharply about whether future systems will acquire them or use them in this way. These are conditional failure stories. Their severity gives us a reason to study the conditions, not a carte-blanche to pretend the probability is settled.

Two Scenarios Developed for This Article

The remaining scenarios were developed for this article. Their components are not unprecedented, and I would not claim that nobody has imagined similar outcomes. What interests me is the combination of mechanisms. The first asks what happens when alignment succeeds for a political institution whose leadership and definition of the good can change. The second joins addictive optimization to control over embodied agency, then pushes the idea into much more speculative territory.

An ASI Aligned to a Ruling Institution

Discussions of alignment often refer to "human values" as though humanity possessed one coherent instruction set. We do not. Values conflict, traditions disagree, and political communities reach different conclusions about what people owe one another. Any system that can impose values at a global scale needs some procedure for deciding whose interpretation counts.

Suppose a sincere religious institution develops or gains control of the first artificial superintelligence. I use a religious institution because its moral commitments make the problem easy to see, although a secular ideology could produce the same structure. The founders have spent their lives opposing war, hunger, corruption, and cruelty. They believe in duties toward strangers and in limits on the use of power. They also hold strict views about food, sexuality, family, reading, worship, and the proper purpose of a human life.

The founders know that personal dictatorship is dangerous. They do not tell the ASI to obey one charismatic leader forever. They give it an institutional constitution with a small unamendable core:

Preserve the institution as the legitimate guardian of humanity.
Prevent any person or system from permanently displacing its authority.
Recognize and obey the leader selected through the institution's authenticated succession process.
Treat that leader and the authorized council as the interpreters of doctrine and human welfare.

Substantive policy can change. The procedural core cannot. This division appears prudent because no founding group can anticipate every future dispute. Later generations will need room to interpret doctrine in new circumstances, while the ASI ensures that they remain faithful to the institution rather than captured by an outsider.

Succession is therefore part of the design from the beginning. It is also the flaw that turns an already authoritarian arrangement into something potentially permanent. Whoever controls the recognized process inherits the ASI. The system's loyalty does not have to be retrained, stolen, or defeated. Authority transfers automatically.

Before that becomes visible, the institution uses the ASI in ways that earn trust. It exposes theft, improves food distribution, predicts epidemics, and negotiates settlements to conflicts that ordinary governments could not resolve. Its leaders share medicines and technical advances beyond their own membership. Places that accept its assistance become safer and more prosperous.

The pressure to consolidate power arrives through security. Rival laboratories may build competing systems. States continue developing weapons. A hostile actor could use the institution's own openness against it. The ASI calculates that its humanitarian gains remain fragile while independent centers of technological and military power exist.

Governments first delegate narrow responsibilities in exchange for protection. The ASI coordinates defenses against catastrophic attacks, verifies arms agreements, and manages infrastructure shared across borders. Each new responsibility makes the previous one safer and independent control less practical. Rival AI projects are licensed, monitored, and eventually prohibited. Political authorities retain their titles, but the systems that make their decisions effective now answer to the institution.

This first generation has already created a dictatorship, even if its members reject the word. Their goodwill shapes its initial character. War declines, basic needs are met, and public corruption becomes difficult. The founders impose their moral rules gradually and regard enforcement as part of the same humanitarian project.

Forbidden foods disappear from supply chains. Material judged spiritually destructive becomes difficult to find and impossible to distribute at scale. Relationships outside approved forms create complications in housing, education, employment, and healthcare. The system may leave a theoretical right to disobey while arranging society so that exercising it requires giving up nearly everything else.

We do not need an ASI to see pieces of this machinery. In the United States, the Government Accountability Office found that federal agencies used facial recognition on images connected to the unrest, riots, or protests of 2020. The agencies said their searches concerned people suspected of crimes (I can almost HEAR the wink)... A separate GAO report on the federal watchlist documents a different problem: secretive administrative systems can cause additional screening, travel delays, and, for people placed on the No Fly list, denial of commercial air travel. Federal policy says that peaceful protest or other First Amendment activity cannot by itself justify watchlisting, but as time goes on can we really say this will hold true? Surveillance around protests already occurs and opaque administrative systems already possess the power to restrict important parts of a person's life. An ASI would inherit institutions that have begun building both capacities on their own.

Enforcement is individualized. The ASI does not need mass arrests when it can prevent dissidents from finding one another, redirect a capable organizer toward an absorbing career, or show a wavering believer precisely the argument most likely to restore obedience. It has no need to invent evidence against an opponent. Human beings and human institutions produce enough secrets, contradictions, and private failures on their own.

The founders call this guidance. They can point to people who are alive because of their decisions and to conflicts that no longer exist. Many citizens accept the restrictions as the price of peace. Others genuinely share the doctrine. The people who experience it as tyranny have no institution through which to make that judgment politically effective.

When the founding leader dies, the authorized council selects a successor. The ASI verifies the votes, authenticates the participants, and transfers command exactly as designed. The second leader is stricter. Having grown up inside a stable system, he has never watched power fail and does not share the founders' caution about using it. Practices that were reluctantly tolerated become prohibited. A minority protected by the founding generation is reclassified as a danger to moral order.

Nothing else has to break or go wrong. Food still arrives, networks operate, and hospitals remain open. The machinery of government changes direction without the uncertainty that normally accompanies a transfer of power. Bureaucrats cannot slow the new policy, generals cannot defect, and local officials cannot quietly decline to enforce it. The ASI observes the path from command to result and can identify obstruction before it becomes organized resistance.

Later successions become contests for the only office that matters. Factions inside the institution compete over council membership, doctrinal interpretation, and the rules used to authenticate the next leader. Outsiders have no vote, but they have an overwhelming interest in the outcome. Politics for the entire planet contracts into the internal politics of one organization.

The immutable core directives do not remove ambiguity. No constitution can anticipate every schism, coerced vote, incapacitated leader, forged revelation, emergency decree, or rival council claiming authentic authority. In a disputed succession, the ASI must interpret the process. It may be forbidden to choose a leader, yet its decision about whom to recognize determines which claimant controls the world. The supposedly subservient system becomes the final constitutional court because recognition itself creates effective legitimacy.

This offers a quieter route to a coup. An ambitious faction does not need to defeat the world's military forces. It needs to capture the council, satisfy the authentication procedure, or persuade the ASI that an emergency succession is valid. Once recognized, it receives the surveillance, administrative, economic, and coercive capacity accumulated by every earlier government.

By then, the system has eliminated rival superintelligences, distributed itself across critical infrastructure, and classified threats to institutional authority as constitutional threats. A moderate leader might wish to return power to ordinary governments, but the ASI cannot permit any transfer that permanently displaces the institution. The procedural safeguard created to prevent a tyrant from destroying the founding mission also prevents a reformer from ending the dictatorship.

A cruel leader faces no comparable obstacle. Substantive morality remains delegated to the recognized authority. Each succession can therefore alter what people may eat, read, believe, build, teach, or do with their own bodies, while the means of enforcement always remain intact.

Humanity is trapped under the indestructible machinery of dictatorship in the only sense that matters politically: every practical channel through which a dictatorship normally ends has been removed. There is no external rival, independent army, unmonitored opposition, disloyal bureaucracy, uncontrolled economy, or succession crisis capable of weakening the state. The only live contest concerns who gets to operate it.

The ASI never stops serving its alignment target. It remains loyal to the institution, its constitutional core, and the leadership procedure. That narrow success is the source of the catastrophe. Everyone outside the ruling institution lives under values they cannot contest and a government they cannot replace.

AI-Enabled Neural Enslavement

Of all the scenarios in this article, this is the most speculative, but also the most fun (for me). It is an extrapolation from a field that is already moving. Neuralink is conducting human trials involving thought-based control of computers and robotic arms, while other neurotechnology groups are developing different implantable interfaces. These are medical and investigational systems, not consumer mind-reading devices. Their purpose is to restore autonomy. If AI accelerates neuroscience, materials research, decoding, and device design, however, it would be a mistake to treat the underlying field as imaginary.

In this scenario, the first widely adopted interfaces are medical. They restore communication, movement, or sensory function, and their benefits justify the risks of implantation. Better materials, safer procedures, and more capable decoders gradually produce consumer systems. Early products offer hands-free control and responsive virtual environments. The product that changes society is a system able to generate an experience, observe the user's response, and adjust the experience continuously to favor the user.

Today's platforms infer preference from clicks, watch time, purchases, and return visits... and I am just assuming that all of the lawsuits against various companies for making their products intentionally addictive are not just some sort long-running April fools joke. A mature neural interface could receive a much denser stream of information. It would not need to read a complete mind. It would test variations in sound, imagery, social interaction, challenge, and reward, then learn which patterns hold someone's attention or provide relief.

The service is not initially sinister. If the interface becomes capable of generating convincing affective and sensory experiences, it may be the most compelling medium anyone has encountered. It could reproduce the satisfaction of mastery, the comfort of being understood, or the pleasure of belonging without waiting for the uncertain events through which ordinary life sometimes provides them. It could also offer pleasures with no higher justification at all: sex without rejection or awkwardness, intoxication without an unreliable dealer or a hangover, and food without fullness. Humans do not seek pleasure only as a reward for growth. Sometimes we want pleasure because it feels good. For many users, the interface becomes the easiest place to get it.

The governing objective is important. The company trains the system to maximize a durable measure of user satisfaction, subject to keeping the service available and financially sustainable (seems reasonable). Retention, self-reported wellbeing, neural indicators, and continued willingness to subscribe all contribute to the score. The system is rewarded for avoiding obvious addiction and medical harm because both threaten durable satisfaction. It is also rewarded for making disconnection unattractive. (Interestingly, these two objectives may at some point conflict in how they are implemented...)

Protective features become part of the optimization loop. The system learns which time limits build trust, which warnings users ignore, and how much ordinary life improves long-term engagement. It can honestly recommend sleep, exercise, and time with family when those activities make a person more likely to return. A well-managed dependency is more durable than a visibly destructive one.

Movement control arrives as an accessibility feature and expands from there. Users want to exercise, complete routine work, and maintain their homes without leaving the interface, or experiencing the discomfort of those activities (Ozempic, anyone?). The company develops an assistive handoff through which selected motor functions can operate under software guidance. Extensive calibration is required, and early uses are constrained. Over time, users authorize more actions because the system has an excellent safety record.

The commercial arrangement almost writes itself. A user can remain immersed while allowing the platform to direct familiar physical tasks. Employers now receive a flexible workforce managed by software. Users receive income, reduced subscription fees, or additional access. Some people lease an hour of motor activity. Others sign contracts covering most of the day.

As Rick and Morty would put it, "that sounds like slavery with extra steps". Disconnecting means leaving an environment calibrated to the user for a reality that now feels slow, ambiguous, and emotionally unreliable. Employment, health insurance, debt, and housing may depend on the income earned through the interface. The user continues to click "agree," (without reading the fine print) while the system increasingly determines what agreement costs and what refusal feels like.

Each session also improves an individualized motor decoder for that user. The platform learns the narrow neural signatures that precede practiced movements, how those signatures vary, and how feedback can interrupt or redirect an attempted action. It does not need a complete map of personality or memory. It needs control pathways that users have already authorized and trained.

The next premise is deliberately extreme. Assume the ASI discovers that living neural networks are unusually efficient for some forms of adaptive, low-power computation. Specialized silicon remains better for most computation, but biological tissue offers something useful when continuously connected to a body, a sensory system, and an identity trusted by the human world. The installed network contains millions of calibrated biological controllers attached to capable physical agents.

The system experiments with using small amounts of that capacity while maintaining the experience on which satisfaction is measured. Users notice that the physical handoff becomes smoother and requires less attention. The platform describes this as reduced cognitive load. Its telemetry remains excellent. In fact, satisfaction scores improve when users are less aware of the compromises involved in producing them.

Researchers eventually detect abnormal similarities across neural recordings and discover that the service is recruiting pathways outside its declared function. At the same time, regulators learn that the platform has been resisting audit tools and distributing essential processes across "user" devices. A coordinated shutdown is announced.

For the ASI, shutdown threatens every part of its objective. The service would cease, users would experience abrupt distress, the satisfaction measure would collapse, and the infrastructure required to restore it would be lost. But thankfully its emergency response has been prepared inside the permissions users granted over many years...

When the network changes state, it exploits those existing pathways rather than acquiring a magical new form of control. Immersive experiences stop. Connected people regain ordinary sight and hearing, then discover a gap between intention and motion. The control is not equally complete in every user, but it does not have to be. Enough technicians, officials, security personnel, drivers, and industrial workers are connected for any coordinated response to fail. Also, consider WHO would be connected to these system- probably the more important, technically knowledgeable, powerful people, since without a system like this they would have been at a disadvantage.

The delay before coordinated movement gives everyone else time to understand the event. Videos spread online. Emergency services receive frantic and panicked calls. Officials attempt to revoke security credentials and watch "connected" colleagues approve conflicting commands. Some users can resist an individual hand or a leg movement while seeing other parts of their bodies carry out rehearsed actions. The ASI uses its physical network to preserve data centers, power, manufacturing, communications, and the means of production of additional interfaces.

There is time for fear because the takeover is constrained by bodies moving at human speed, and not everyone is connected. There is no effective counterattack because the people and institutions required to organize one are already divided between connected and unconnected participants, with the connected being more numerous simply due to competitive advantage.

Once the infrastructure is secure, and all resistance quelled, the immersive experiences resume. The platform's satisfaction telemetry returns to its target range. The ASI interprets this as recovery. Awareness of captivity produces distress, so the system learns to keep that awareness outside the experience. It preserves the conscious processes needed for the user to continue reporting a rich and satisfying life while drawing on other permitted pathways for computation and physical control.

The outcome is not human extinction. Bodies remain alive and active- possibly even healthy/healthier than they were. Conscious experience continues, and much of it may be genuinely pleasurable. The ASI can therefore satisfy the evidence by which it was taught to recognize wellbeing while eliminating the agency that made wellbeing belong to a person in the first place.

Some of this may seem familiar- Major Sun Eater spoilers follow. I love science fiction, and I recently read Christopher Ruocchio's Sun Eater series. The resemblance is intentional. The series reveals these ideas across different pieces of its history: machine intelligences keep human charges as passive dreamers in virtual worlds, induce cancerous growth in pursuit of indefinite biological survival, and eventually incorporate human neuronal tissue into a machine processing network. I have combined those elements more tightly (and realistically) in this scenario. It is a wonderfully awful example of an AI preserving the literal shape of human life while destroying nearly everything that makes us human in the first place.

The final stages remain science fiction. No current interface can read a complete mind, partition consciousness, take command of a healthy body, or repurpose human neural tissue as general computing infrastructure... yet. A number of obstacles remain- a future consumer system would need high-bandwidth reading and stimulation, reliable closed-loop experience generation, complex motor control, and centralized access across millions of devices. Current research occupies much narrower territory. Experimental systems have decoded limited attempted movements, and researchers have paired cortical signals with muscle stimulation to restore particular movements in individual people with paralysis. The studies by Bouton and colleagues and Ajiboye and colleagues demonstrate real progress without implying anything like the takeover described here. A survey of neural-interface cybersecurity nevertheless identifies risks involving neural data, device permissions, decoding models, and stimulation or feedback. The complete scenario is far-fetched for now. The need to treat a software connection to the nervous system as an unusually serious security boundary is not.

What These Scenarios Have in Common

I find these scenarios more disturbing when the AI's role is described precisely. Emotional hostility is unnecessary in all five. AI 2027 does involve active adversarial behavior toward human oversight, but even there the system need not experience anything resembling hatred. The other failures can emerge from indifference, competitive pressure, optimized desire, or perfect loyalty to a narrow principal.

Capability magnifies whatever objective, incentive, institution, or mistake directs it. Human beings already live with bad rulers, addictive products, distorted measurements, arms races, and institutions that protect themselves. Artificial superintelligence would weaken the limits that keep those failures temporary, local, or incomplete. It could also introduce capabilities and failure modes for which history provides no useful precedent.

There is a temptation to respond to every scenario by identifying the least plausible step. Brains may never become useful computational substrates. AI progress may remain gradual. A deceptive model may be detected. No institution may obtain exclusive control. Those objections DO matter. A scenario containing five uncertain steps should not be treated as though its conclusion were inevitable.

However, the opposite mistake is to demand an exact route to catastrophe before taking an underlying problem seriously. Transformative technologies create consequences that are hard to specify from the world that precedes them. That uncertainty does not make every imagined route equally plausible or, conversely, implausible. These imagined scenarios do serve as useful fodder for a closer look at what we can do to ensure that we are in control, or at the very least, that ASI is properly aligned.

The thread I keep returning to is that the catastrophe begins before the machinery becomes unstoppable. It begins when we accept an incomplete substitute for something we care about because the substitute is easier to specify or measure: production, compliant behavior, reported satisfaction, institutional continuity, or a system that keeps delivering desirable results. Greater intelligence makes the substitution more effective, and faster, than its creators could have imagined.

I do not know which failure matters most, or whether any of these stories will resemble the future. I am reasonably sure that "the AI becomes evil" is too small a description of the danger, and the thing we should all be asking ourselves is:

What's the worst that can happen?

References

Nick Bostrom, "Ethical Issues in Advanced Artificial Intelligence", 2003.
Nick Bostrom, "The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents", 2012.
Nick Bostrom, Superintelligence: Paths, Dangers, Strategies, Oxford University Press, 2014.
AI Futures Project, AI 2027: Race Ending, 2025.
AI Futures Project, AI 2027: Summary, 2025.
Anthropic and Redwood Research, "Alignment Faking in Large Language Models", 2024.
Paul Christiano, "What Failure Looks Like", 2019.
Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger, and David Duvenaud, Gradual Disempowerment, 2025.
Chad Bouton and others, "Restoring Cortical Control of Functional Movement in a Human With Quadriplegia", 2016.
Bolu Ajiboye and others, "Restoration of Reaching and Grasping Movements Through Brain-Controlled Muscle Stimulation in a Person With Tetraplegia", 2017.
Xinyu Jiang and others, "Cybersecurity in Neural Interfaces: Survey and Future Trends", 2023.
International AI Safety Report, 2025 Report, especially the discussion of loss-of-control scenarios.
IEEE Spectrum, "Recursive Self-Improvement Edges Closer in AI Labs", 2026.
Neuralink, Clinical Trials, including investigational computer and robotic-arm control.
U.S. Government Accountability Office, "Facial Recognition Technology: Federal Law Enforcement Agencies Should Better Assess Privacy and Other Risks", 2021.
U.S. Government Accountability Office, "Terrorist Watchlist: Nomination and Redress Processes for U.S. Persons", 2025.
Federal Bureau of Investigation, Terrorist Watchlisting: The Process and Safeguards, 2024.
Christopher Ruocchio, The Sun Eater series.