Forethought

One of the most important questions facing the world is whether AI will undergo a fast intelligence explosion and quickly automate most real-world work. A common argument against a fast intelligence explosion is data bottlenecks: data is a crucial input to AI training, and how could it increase fast enough to fuel such explosive progress? This paper attempts to identify strong data bottlenecks, but we’re ultimately sceptical that any such bottlenecks would stop an intelligence explosion from accelerating.

The long-term future of intelligent life is currently unpredictable and undetermined. We argue that the invention of artificial general intelligence (AGI) could change this by making extreme types of lock-in technologically feasible. In particular, we argue that AGI would make it technologically feasible to (i) perfectly preserve nuanced specifications of a wide variety of values or goals far into the future, and (ii) develop AGI-based institutions that would (with high probability) competently pursue any such values for at least millions, and plausibly trillions, of years.

Governments will deploy frontier AI in areas where their powers are already least checked: military, intelligence, surveillance, and policing. Companies can't police those deployments, and legislatures largely can't see them. This could undermine democratic checks and balances, and even enable coups: in particular, as AI replaces human bureaucrats and soldiers, there might be nobody left to refuse unlawful orders. This piece lays out several kinds of work to mitigate these risks: legislating rules for AI in government, interbranch oversight, model specs for systems designed specifically for government contexts, and strengthening civil society.

We make the case for training AIs to be risk-averse in resources — specifically, to treat resources as having diminishing marginal utility. We argue that risk aversion can preserve AIs’ usefulness in the event that they turn out aligned, and that it provides an extra line of defense in the event that AIs turn out misaligned: misaligned but risk-averse AIs would prefer a higher chance of modest payments to a lower chance of successful rebellion, so in many circumstances we could pay these AIs not to rebel against us. We sketch out some possible methods of training AIs to be risk-averse, and we give reasons to be cautiously optimistic about these methods’ success. Overall, risk aversion seems like a promising line of defense against threats from misaligned AI.

AI companies face a tangle of competing considerations when deciding what goes into a model spec. Should the AI be completely honest in every circumstance? Take proactive prosocial actions? Engage in whistleblowing? And so on. We lay out a checklist of plausible criteria for good model spec design within four categories: behavioral usefulness; accountability and evaluability; coordination and common knowledge; and trainability and LLM psychology.