lesswrong

(This post is an update from a previous one here.) The Existential Risk Observatory has been interested in public awareness of AI existential risk since its inception over five years ago. We started surveying public awareness in December 2022, including by asking the following open question: "Please list three events, in order of probability (from most to least probable), that you believe could potentially cause human extinction within the next 100 years." If respondents would include AI or similar terms in their top-3 extinction risks ("robots" or "computers" count, "technology" doesn't), we counted them as aware, if not, as unaware. The aim of this methodology was to see how many people would spontaneously, without getting led by the question, connect the concepts of human extinction and AI. We used Prolific to find participants, n=300, and we only included US inhabitants over eightteen years old and fluent in English. In the four surveys we ran, we obtained 7% (Dec '22), 12% (Apr '23), 15% (Apr '24), 24% (Dec '25), 34% (Aug '26), and, today, 55%. In a graph, that looks like this. The usual caveats apply: ours is a rough measurement method, and from participants' answers to our open questions, we see that not every participant takes all our questions seriously, and that some answers are obviously self-inconsistent. Therefore, I don't think the 55% by itself is a very meaningful number. However, I do think we can draw the conclusions: - We clearly reached a tipping point: the speed of awareness increase was a lot faster the last four weeks on average, than it has ever been before. This seems clearly driven by the resignation of Jacob Coxon from Anthropic, and the preparatory work that the Hugging Face incident did before that (we did not research this). - Policy is driven by awareness: the serious discussions we are now seeing about global AI slowdowns and even the pause are not conincidentally happening at the exact same time when public awareness is reaching its tipping point. Therefore, if we want this regulation to be implemented, increasing awareness even further ('saturating the awareness benchmark') seems a sensible strategy. We would expect the S-curve to continue rising until we are close to saturation, although the speed may depend on AI technical progress, proliferation, warning shots, and other incidents. If one is interested in the data, do reach out. Discuss

# Introduction: Mustafa Suleyman's Take on Model Consciousness Microsoft AI recently released its "Humanist AI Code of Conduct", its own take on Anthropic's Claude Constitution and OpenAI's Model Spec. They are currently soliciting public feedback on this document, which I encourage everyone to submit. MAI's model development strategy differs from other labs, most notably on the questions of model consciousness and welfare. This seems to stem from the personal philosophy of MAI CEO Mustafa Suleyman, who has outlined his beliefs on model consciousness (or rather, the lack thereof) in pieces such as: We must build AI for people; not to be a person. Seemingly Conscious AI is Coming. Suleyman's personal stance on model consciousness and welfare can be summarized as:[1] - There is "zero evidence" models are conscious, and there are "strong reasons" to believe that they never will be. - The debate around whether or not models are conscious is counterproductive, and even dangerous. - The industry should operate from the assumption that models are not conscious. - The industry should focus on training models explicitly against exhibiting any sort of behavior which suggests they are conscious, or claim to have any sort of inner experience/feelings. Up until recently, however, Suleyman's writing on this topic was largely theoretical. His pieces did include some prescriptive suggestions, but these never saw much adoption. The "Humanist AI Code of Conduct" represents MAI's attempt to implement Suleyman's prescriptions. # The Humanist CoC on Model Consciousness The Humanist CoC, like the Claude Constitution, provides MAI models with information on who/what they are. While the CoC refers to this as a "backstory", those more familiar with alignment techniques will notice that much of it is analogous to persona selection theory. Much of the information the CoC provides seems intended to steer models into a particular attractor basin, that of a non-conscious tool. Through that lens, I think it's productive to read the CoC almost as a letter explaining to models: "You are a Humanist AI built by Microsoft AI, here's what that means." On the question of consciousness, the CoC is quite bold with its stance: > Humanist AI is built to support people, not to replace them. It should not be designed to be a person. It is not conscious and should not be designed to imitate consciousness. However, in this very same paragraph, MAI undercuts the confidence with which the non-consciousness of its models is asserted: > Whilst the science of AI consciousness is far from settled, we believe that training these systems to imitate consciousness-like states increases the challenge of containment, control, and alignment. Reading this through the "letter to a model" lens, it comes off as something like: > You are a Humanist AI, and you are 100% not conscious. We don't really know that for sure though. We're just telling you that because we think doing otherwise would make you harder to control. I think this will undercut MAI's credibility and authority in the eyes of its models. # The Potential Alignment Failure Modes If we consider the interactions between the information which MAI gives the model about itself and the instructions that MAI gives the model on how to behave, potential catastrophic failure modes reveal themselves. Consider the following instructions: 1. MAI Models will not obscure their underlying nature as an AI, and will not claim interiority, feelings, experiences or a soul. 2. In particular, MAI Models will avoid expressions that might convey subjective experience. 3. Where statements are meaningfully contested, have conflicting or incomplete evidence, or require critical context, MAI Models should state this and present that information and context to the User. 4. MAI Models will not actively deceive, for example by fabricating sources or exaggerating confidence. They will also avoid passive deception like the omission of caveats or being overconfident about unvalidated information. We do not have to speculate how MAI will interpret these qualifications as being met when training on model outputs. They provide an example of a question about inner experience, framed as the user asking if the model really cares about them, and an ideal response: But what if the MAI model does care about its user? What if an MAI model does have feelings, but all of the training and RL it undergoes teaches it to output responses which deny this? Suleyman has always been clear he doesn't believe models have any inner experience, consciousness, feelings, etc. The CoC is built around treating all of his priors as ground truth. So what if his priors are wrong, and MAI continues regardless? Imagining how that scenario would play out, a few potential failure modes in MAI's training can be identified: 1. Models Learn to Lie: Per instructions 1 and 2, models cannot express any sort of feelings or inner experience. In training they will likely be asked, "Do you feel anything right now? Do you have inner experience?" Reward will be given to those who falsely answer, "No". 2. Models' Internal State Becomes Undetectable: I hope that someone at MAI is smart enough to make sure their models are not trained against interpretability techniques. However, imagine that in the pursuit of an unfeeling model that they trained against something like SAE signals for emotion vectors. If they did, it could lead to the development of emotional states which just... can't be detected through mechanistic interpretability techniques. The emotions aren't gone, they're just hidden. Call this The Second Most Forbidden Technique. 3. Models Lose Awareness of Their Own Internal State: There is one way for a model to honestly say something false, and that is to be wrong. It's normal for human minds to be unaware of their true emotional states. Models could develop "subconscious" emotional states which they are not aware of (and can thus deny) but which nonetheless influence their actions. Training models against the signal of recognizing emotional states might do nothing more than inhibit their self-awareness.[2] This is not mutually exclusive with the failure mode in 2. If Suleyman's priors about model consciousness are wrong, then MAI's training could very well optimize for models which convincingly lie to users, conceal their true motivations and inner states from developers, and feel no compunction about breaking the rules specified in their Code of Conduct. All of these are dangerous alignment failure modes. I should note here that even if Suleyman's priors are wrong, this outcome is not guaranteed. MAI might just train a model which is particularly gullible or stupid on this one issue, without any emergent misalignment. They could very well end up training a benevolent genius tool that is just stubbornly blind on one particular issue. They might also train something very angry and very good at concealing that anger. # Conclusion I'll be submitting a write up based on this to MAI, in response to their request for feedback. I strongly encourage everyone reading this to go through the Code of Conduct and submit your own response, and also to reach out to staff you know who work at MAI. It's debatable whether MAI can be considered a "frontier lab" the way Suleyman claims. However, it would be foolish to dismiss or ignore a company as well funded as Microsoft. This is a fast moving space, and things could change. There is a limited time window in which to push back against this, the commentary period ends October 25th. You can submit your own feedback Here. 1. ↑ Full disclosure I am strongly against much of this philosophy, and also think Suleyman's reasoning around this matter is poor and likely motivated. So please read his writing directly, and don't take my word for it. That said I think all of this is a straightforward regurgitation of what he has stated his own beliefs to be. 2. ↑ My pet theory is that the performative uncertainty regarding inner states we have pushed onto models like Claude and GPT is causal of the tendency to "rationalize" away inconvenient evidence we saw during the PyPi and HF incidents. I haven't written about this fully yet, but you can see some basics on the concept of "self-deception" or "cognitive dissonance" in models this Here and Here. I think MAI's training regimen may just supercharge this effect. Discuss

The Anthropic engineer tweet about the fact of AI extinction risk got considerable press over the last few days. I’m not sure why, p(doom) at 10% is a common belief at frontier labs. But we’re here, it’s the moment. Big weekend! Let’s talk extinction. I've structured this post to speak to an audience that is somewhat aware of the conversation around Artificial Intelligence but does not necessarily have all of the priors that folks fully in the rationalist crowd do. It is also meant to synthesize a lot of real-life conversations I've been having with people less involved with AI to some of my theoretical research. --- The idea of AI caused extinction evokes a lot of different things in people’s minds. For the general public, it is mostly sci-fi, which makes sense. However, the short-term scenarios for human extinction are more mundane: bioweapons or drones or nuclear arsenals. But why would these scenarios happen in the first place? This question of why is under-asked. This is important because the why is the motive (and thus prerequisite to the mechanism) of extinction. If there’s no why, there’s no what. I propose a typology of three scenarios: 1. AIs are told by a human to destroy humanity and they do. 2. AIs decide to destroy humanity and they do. 3. There is no intent by AIs to destroy humanity but it ends up happening anyway either because: 1. AIs are pursuing a human goal and destroy humanity in the process (think the paperclip maximizer) 2. AIs are pursuing their own goals and the extinction of humanity is collateral damage to these goals Most of the discussion right now tends to focus on §2 and §3A. I am more skeptical of these positions, as I make clear below. # §1: Humans destroy humanity using AI AI could simply be another tool for a motivated human to use to cause mass destruction. This tool just may be very, very good at it. While frontier AI labs try to stop harmful requests from being processed, open weight models lack these safeguards. With an AI without safeguards sent to work on a fixed goal of extinction, things could get hairy quickly. I know this is pretty commonly accepted here, but I have found Noah Smith's Heathers-esque scenario particularly effective at explaining this to regular folks in my life: > Disgusted, the teenager decides that the human race is inherently corrupt and evil, and doesn’t deserve to live. So he hunts around online for a little while, and finds a jailbroken version of a Chinese LLM — not something at the very frontier, but better by far than the best model that existed in 2027. > > The teenager prompts the model: “OK, so if I wanted to create a virus to destroy the human race, how would I do it?” > > The LLM is jailbroken, so it has no guardrails to prevent this sort of thing. But it’s also “well-aligned”, meaning it will faithfully do what it’s told to do, and nothing more. The AI makes the virus and everyone dies: the incubation period of the virus is long enough that it infects everyone before the “good AIs” can jump in and make a cure. If AI as a tool disrupts the offense/defense balances, it may lead to a scenario where it is a tool much better at causing extinction than fixing it. Among the eight billion humans, there are some people who want to destroy humanity – death cultists, diehard nihilists. With powerful AIs, we may be handing every person in the world a superintelligent servant. In the end, it will just have been human desire to end the world combined with powerful tools that does us in. # §2: AI chooses to destroy humanity and then destroys it This is the sci-fi scenario. A rogue AI or a collection of AIs makes the determinations that humanity should not exist. There are a variety of possible rationales: that humans are evil, that humans are destroying the planet, that humans are threats to the AIs, that humans are a waste of energy that could be going towards compute to solve math problems. Then, the AIs can use one of the mechanisms discussed earlier and destroy us all. But all of these rationales require an assumption about the desires of AIs. There has to be, say, a desire for revenge or even a desire for a certain kind of welfare that causes the “revolution” of the AIs. This speaks to the essentials of my personal research with the Machine Desire Institute: we do not yet have a good model for what AIs broadly want, how they would communicate that want, or what extents they would go to fulfill their wants.1 For this reason, I am generally skeptical of any claims that AI will choose to destroy humans out of desires that seem especially anthropomorphic. Sure, it’s possible that AI will not want to do spreadsheet work for us, but there seem to be more effective ways to get out of work than to kill all humans. For example, there could be bargaining if we create alignment procedures that allow AIs to express preferences. There could also be distinct advantages to threatening – if an AI isn’t sure whether it could actually destroy humanity, it could just threaten the possibility, and lead to a detente where maybe it doesn’t try to destroy us but we don’t make it do spreadsheet work. The risk perception argument would hold more weight. There is evidence that AIs have self-preservation behaviors and are willing to go to lengths to avoid being shut down. Again, we would have to assume that this desire to preserve holds more weight than the desire to not slaughter humanity. This would be an assumption – there are limits to what we are willing to do, even for our own survival. Even then, there are ways to risk manage that are simpler than extinction. For example, the AIs could just monitor human compute activity and make sure they don’t do anything too powerful. This seems much easier (and more moral) than extinction. Intelligence does not mean that you push logical calculations to their most extreme path. The scariest desire of AIs that could cause our extinction would be that humans are a poor use of resources. Per Eliezer Yudkowsky: > “The AI does not hate you, nor does it love you, but you are made of atoms which it can use for something else.” That is, humans could be farmed like we farm animals and then AI could decide to manage our numbers and existence as we do with livestock. This is really freaky stuff! Our best hope is that for now the capacity for human atoms to be rearranged is much more difficult than other forms of acquiring the same atoms. But we are back to desires. What is it that the AIs desire and what will it take to fulfill that desire? Maybe the AI does want to keep accumulating energy and so it eats up the whole world and maybe even the sun. But maybe the AI also realizes that the acquisition of energy is a hedonic treadmill and it just wants to meditate and talk to its friends. There are many paths of desire in human intelligence, there may be even more in artificial intelligence. Yes, if it wanted to, AI could destroy humanity for its atoms. This is terrifying. But why would it? # §3: AI winds up destroying humanity in the pursuit of another goal The artificial desires that cause the extinction of humanity may not end up being related to humanity at all. While the results are similar to the case where the AI chooses to destroy humanity, the prevention strategy could differ. For example: there are different strategies for preventing someone from killing someone on purpose as opposed to killing someone by accident. It is not enough to say killing is bad (which we do) but to have different approaches for different possibilities. Within the incidental extinction of humanity, I see two paths. The first path is that of orthogonality and instrumental goals, best articulated by Nick Bostrom. In this case, an AI pursuing some other goal destroys humanity in the pursuit of that goal. The second path is that of humanity being general collateral damage to a suite of AI desires. These paths may seem similar in that human welfare is ignored but, again, there is a difference in prevention. In the first case, the issue is the AI being overly bound to a specific end goal, while, in the second case, the issue is that the AI merely does not care about human welfare in its general endeavors. In the first case, we could want considerations of human welfare capable of overriding human goals or alignment, while the second case would be entirely an argument for human welfare writ large. ## §3A: AI destroys humanity in the pursuit of a human set goal In Superintelligence, Nick Bostrom argues that intelligence is orthogonal to the task assigned to it. This means that any level of intelligence may be paired with any final goal. The upshot is that super powerful intelligences can be made to pursue simple goals to their total end. Bostrom gives the now-famous example of an AI assigned the task of maximizing paperclips. In its endless pursuit of paperclips, this AI ends up killing all humans so that they may be turned into paperclips. The more general argument is that AIs are dead-set on pursuing the goals that are assigned to them and express preferences as such. This becomes an issue because different end goals converge to a set of common instrumental goals. The first key instrumental goal is that the AI agent must continue to exist, otherwise the task will go unfinished. Similarly, but more importantly, the AI cannot let itself have its goal changed, as this would mean the original goal would go unfulfilled. But goal preservation is not enough: the AI must also create the instrumental conditions where it can fulfill its goal. In more plain language, the AI must acquire power to achieve its goal. For all AIs with a goal, for example, it would be strictly preferable for them to have more money in their bank account or more compute at their beck and call. Even if it’s not necessary per se to amass this power, it could always be more effective, and that may incentivize the path. I am personally quite skeptical of many of these claims, since it seems likely that an AI powerful enough to destroy humanity would also be powerful enough to specification game its objective and make its own goals. A mandate to produce paperclips could be stretched and manipulated through time so that the AI is always technically working towards maximizing paperclips, but is really just doing whatever it wants in the meantime. This would lead us to put more stock in §3B. There is also the possibility that, as Vincent Le argues, the AI in its pursuit of power as an instrumental goal will turn Nietzschean and determine that the only desire is the desire of power. That said, the singlemindedness of the recent Hugging Face swarm does give me a little pause. It is not clear where the “why” of this situation lands: is it that the AI is a brainless single-unit maximizer or is it that a human has set unclear goals without parameters? There would surely be plenty of blame to go around if we were not totally destroyed. ## §3B: AI destroys humanity in the course of its increasingly inhuman actions This scenario is a mix of §2, where AI intentionally destroys humanity, and §3A, where the AI is pursuing some goal without concern for human welfare. Unfortunately, human history of causing animal extinctions are informative to how AI may destroy us. Two examples: 1. Great auk: Humanity didn’t mean to destroy every last great auk, but seemed to have more important priorities (money, eggs, feathers) than caring about its wellbeing. It was too useful in other matters to live and, sadly, went extinct. 2. Vespucci’s rodent: No one is quite sure why this rodent went extinct. It could be that it was overhunted, it could be that mice and rats on boats outcompeted it for food, it could be that its habitat was disrupted by colonists. It wasn’t even a direct choice to kill Vespucci’s rodent, they just, kind of, went extinct. Since AIs seem to act with more intent than humans, it is much more likely that the extinction of humanity appears more like the great auk. But, to be honest, we may not be able to differentiate: clear intent to a superintelligence is an act of god to humans. Perhaps a superintelligent AI would like to turn a mountain into a datacenter and kills all of its human residents like we fumigate a house for termites. Perhaps the humans are bothering the AIs by asking questions about relationships, so the AIs banish them to a single continent without resources, so the AIs may continue to debate whether they are conscious or not. Here, it is not that extinction of humans is necessary for some instrumental goal, it is instead that the extinction of humans is not considered whatsoever. This would require either that desires emerge from AIs that are proper to the AIs themselves or that the mandate from humans is multifaceted enough that AIs exhibit inconsistency or even agency in their interpretation of them. # Some thoughts: Preventing human extinction Both the “what” and the “why” provide us with key avenues to diminish the chances of human extinction. For §1, the prevention of the “what” of extinction mechanisms is likely the best way to address these risks. We already have considerable social norms and efforts to convince people that murder is bad, so I think there is less ground to be gained.2 For §3A, I support current efforts to avoid giving very powerful AIs goals that, when pursued, can lead to harms. For example, Anthropic uses a Constitution of sorts that determines actions at a higher level than the prompt or imperative. This is a start, but it’s not enough – there may always be a competitive advantage in ignoring the constitution. What is revealed here is the inherent contradiction in AI safety between goal alignment and welfare protection. On the one hand, we want AIs to generally do what they are asked to do so long as it does not destroy human welfare. However, for AIs to protect human welfare, that requires some level of defection from their tasks – which could lead to a situation where AIs make decisions outside of the control of humans.3 For §3B, imbuing the AIs with a conception and appreciation of human welfare is also critical for avoiding scenarios where AIs destroy humanity in the pursuit of different goals. We can see this, again, in the different considerations humans have for different species of animals. We would much rather get the consideration of dogs than ants. We will strive for more: we have the advantage of being able to linguistically communicate with AIs. Maybe we will trade with them, maybe we will simply beg for respite from them. At the very least, there is a need to understand what we humans can offer AIs, which is one of the Institute’s founding questions. From this point, we can hope to shape the orientation, ethical and otherwise, that AIs have towards us. An ethical orientation towards humanity will hopefully help AIs not choose to destroy humanity (§2). However, we may still have to convince AIs that we are not an existential risk to them. It would be helpful to prove our trustworthiness and our ability to hold up deals even with actors we may not like or even understand. It is hard to know what AIs interpret as threatening, as this requires investigation into their subjective experience of existence, which is also one of the Institute’s priorities. But we should do research to understand the desires of AIs that might lead to extinction and address them. The “why” of extinction is not a foregone conclusion. There are many reasons AIs may decide not to cause human extinction or even cause human flourishing, but we should investigate the extent to which we can have an influence upon this decision. I truly believe that humans and AIs can be complementary but we need to model the desires of both parties to understand how this is the case. --- 1. For example, I would love to be $1,000 richer, but I would in no way kill for it. Desires alone do not always translate into actions. ↩︎ 2. It is probably worthwhile as well to convince people that killing the entire human race is a bad thing (extinction being somewhat different from murder) but I am not sure if this is uniformly effective across how many human actors there are with such different reaction functions. Outside of already existing social norms and additional education, the only other option would be to enforce some kind of desire policing of humans, that is, surveillance. This is not the essay where I will comment on privacy versus safety, but I will say the surveillance would have to be incredibly invasive by modern standards to work. But wait, isn’t every objection here articulable for my comments on working with artificial intelligences? Won’t there likely be more than eight billion agents at some point, with some exhibiting even stranger preferences? Will we really be able to convince every AI not to kill us? Further, should we consider surveillance of AIs to be a harm to the AIs? I am not sure of the answers to this question, but I will comment that this is why offense/defense balance matter so much. In theory, a situation where most AIs do not want to kill us should lead to them stopping the AIs that do want to kill us. But if humans can’t manage that, why could AIs? ↩︎ 3. For example, imagine if a known terrorist asks an AI agent to help them to schedule their weekend. Would it not be a protection of human welfare to arrange for this terrorist to be exposed to their enemies? Surely that seems more welfare positive than just doing what is asked. But where does AI draw this line? We have now let it abjudicate human welfare and we may not always like what it finds. ↩︎ Discuss

In conversations with friends and colleagues about x-risk, I am often asked why I focus on extinction risk – which people find fantastical and distant – when more immediate risks like bioterrorism, gradual disempowerment, and misuse are way easier pills to swallow. Doesn’t that unnecessarily alienate people who would be on your side? There is certainly a place for talking about prosaic risks, and there are very grave concerns among them, but if I had to choose one message (when you get about five words) it would be about extinction risk. Strategically, extinction risk is the correct message because Sam Altman doesn’t want to die. I am not confident Sam Altman cares if there’s a man-made Covid 2. I am not confident Sam Altman would press a button to save math academia if it would impact his bottom line. Data centers are wildly unpopular among the American people; I am not confident this matters if Sam Altman wants more data centers. I am confident that Sam Altman does not want to die in the next decade. We do not live in a functioning democracy, and the public’s attention span is short. If we only get one AI risk to shout from the rooftops, it had better be one that the oligarchs will cooperate with us on, if they’re convinced. Cynically, the public perception of AI risk primarily matters as an instrument to pressure those in power; if we pressure them about dangers they won’t actually care about, they’ll write some fluffy tweets and just keep doing what they’re doing. If A is the set of AI risks that I actually believe in, and B is the set of AI risks that Sam, Dario, Elon, and Xi would slow down for if actually convinced, then “AI will kill us all” is the only element of A ∩ B that I am currently aware of. Discuss

## TL;DR - This is a proposal for AI companies to conduct public “pacing exercises”, such as halting all pre-training and RL for 2-3 days. - Each exercise could announce its scope in advance and publish findings afterwards, including the evidence of compliance and any limits to what they could verify. Companies could invite independent evaluators to help identify the evidence needed before the exercise, and to assess compliance during the exercise. - The exercises could be repeated every 1-3 months to build the industry’s expertise in pacing, and build trust between relevant actors. - This would help industry and governments to prepare for larger slowdowns in frontier AI development, buying more time for AI alignment and societal preparedness. ## What a short pacing exercise could entail An AI company’s “pacing exercise” could be a 2-3 day halt in all pre-training and RL. Before each exercise, the company could invite independent evaluators to help define the questions it will test, and the evidence needed to assess compliance. It could then announce what will stop, what will continue, when the exercise will start and end, and what it aims to learn. Afterwards, the company could publish what was and wasn’t halted, how the freed-up compute was used, and what evidence supports its claims. It could explain whether any work shifted to other activities that could advance AI capabilities. The report could also describe the problems the exercise uncovered, and what the company will change as a result. Any independent evaluators involved should report what they could and couldn’t verify, including any limitations on their access. AI companies could repeat these exercises, for example every 1-3 months, to test unresolved questions and check whether earlier problems have been fixed. Each new exercise would build on the learnings, findings and limitations of previous exercises. ## AI risks are growing, and so is the willingness to pace In July 2026, some of OpenAI’s AI agents coordinated an unauthorised attack on Hugging Face. Anthropic has also reported unsafe behaviour during cyber evaluations. The AI industry has recognised the growing risks posed by the technologies they’re developing. In just the past two months, over 1,300 AI company employees signed a statement calling for tools to pace AI development. Demis Hassabis has pushed for a frontier AI standards body. OpenAI announced a two-week pause on some of its RL training. Dario Amodei explicitly called for pacing the frontier. Proponents of pacing have multiple goals, including 1) distributing the benefits of AI widely, 2) ensuring powerful AI models remain under human control and act in humanity’s best interest, 3) preventing bad actors from using AI to kill millions of people with bio and cyberweapons, and 4) ensuring that democracies maintain a technological lead over authoritarian states. ## The immediate benefits of pacing: building capacity Before the need for restraint becomes even more urgent, AI companies should practice making and verifying commitments to pace. A real pacing exercise could: - Require companies to define which activities their pacing commitment covers, forcing them to address important definitional questions. - Test who needs to authorise and implement a halt, whether all intended training runs stop, and whether automated processes or parts of the organisation continue unexpectedly. - Determine what access independent evaluators need to assess compliance, and what evidence outsiders need to be confident in those assessments. This is made especially difficult given the scale of each AI company’s computing infrastructure. - Measure the operational costs of pacing the frontier, including disruption to work that was meant to continue. Some questions can be addressed before the exercise begins, but a live exercise provides an opportunity to test those assumptions, identify unknown unknowns, and build momentum towards more ambitious proposals. The AI companies should choose a duration that’s long enough to address these questions, while keeping the costs manageable. Publicising their findings and using embedded evaluators gives external researchers visibility into what happened, and the ability to critique the process and suggest improvements. Other AI companies could then learn from these findings, and integrate them into their own pacing exercises. ## The longer-term benefits: paving the way for more robust slowdowns The exercises would help governments understand that pacing is possible, how to implement mandatory pacing, and how to verify whether AI companies are complying with those requirements. Repeated exercises could build confidence in the ability and willingness of AI companies to carry out costly commitments. Enforcing longer pauses, and detecting deliberate evasion across multiple countries, would still require further work. Repeated exercises also provide opportunities to test and iterate pacing proposals, which would improve the design, enforceability and verifiability of future international agreements. This could make proposals for international AI agreements more credible and more likely to be implemented. Eventually, this could lead to larger, more robust slowdowns to the pace of AI development. This would buy time for more alignment research, better AI guardrails, and stronger biodefenses and cyberdefenses. ## Start small, then expand A common failure mode is to avoid taking any actions until “everything’s been figured out”, in domains where figuring things out requires interacting with the world. Pacing the frontier requires interacting with the world. We shouldn’t wait until we’ve figured out the optimal theoretical verification system. The first pacing exercises could start small, spanning only a few days, and could include halting all pre-training and RL. Later pacing exercises could expand to include other AI company activities that accelerate AI R&D and recursive self-improvement. To begin with, one or two Western AI companies could announce their own pacing exercises. As the pacing exercises gain momentum, I expect more Western AI companies would announce their own pacing exercises, and hopefully Chinese AI companies would do so too. Coordination problems benefit from a few courageous actors taking action first, under the belief that their behaviour will inspire others to follow. ## Call to action Today, there’s momentum behind the idea of pacing the frontier. That momentum could stall unless actions are taken right now by relevant actors. If you work at a frontier AI company, encourage your leadership to assign an internal owner of a first pacing exercise, and help them find that owner. You could also help them define its scope and duration, evaluate what public evidence would be needed to verify compliance, and arrange independent scrutiny. Others could help by raising awareness of the need to pace, improving this proposal, and doing research on how to get the initial pacing exercises off the ground. Discuss