{"id":34696,"date":"2025-03-14T18:28:20","date_gmt":"2025-03-14T22:28:20","guid":{"rendered":"https:\/\/www.thenewatlantis.com\/?post_type=article&#038;p=34696"},"modified":"2025-05-28T09:22:37","modified_gmt":"2025-05-28T13:22:37","slug":"situational-awareness-about-the-coming-agi","status":"publish","type":"article","link":"https:\/\/www.thenewatlantis.com\/publications\/situational-awareness-about-the-coming-agi","title":{"rendered":"Gaining Situational Awareness About the Coming Artificial General Intelligence"},"content":{"rendered":"\n<h4 class=\"has-text-align-center\">Part 1 of \u201c<a href=\"\/publications\/will-ai-be-alive\" target=\"_blank\" rel=\"noreferrer noopener\">Will AI Be Alive?<\/a>\u201d<\/h4>\n\n\n\n<div style=\"height:50px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<p class=\"has-drop-cap\">\u201cBurner of ashes\u201d \u2014 there\u2019s a job description one rarely sees these days. Yet going back to the Bronze Age, the ash burner had an essential if dirty job. Soaking wood ashes and water in a pot, then filtering the liquid and boiling it to evaporate all the water yields potash, useful in making dyes, soap, glass, and fertilizer. From burnt wood to fertilizer: out of the ashes comes new life.<\/p>\n\n\n<div class=\"font-callunasans text-base lazyblock-general-highlight-Z2hQ9eS font-callunasans text-base wp-block-lazyblock-general-highlight\"><div class=\"block-tna-highlight block-offset-float print:hidden\">\r\n\t<div class=\"py-8 px-6 text-center bg-almost-white\">\r\n\t  \t\t<p><a href=\"\/publications\/will-ai-be-alive\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-34568 size-medium\" src=\"https:\/\/www.thenewatlantis.com\/wp-content\/uploads\/2025\/03\/Boyd-banner2-1280x569.jpg\" alt=\"\" width=\"1280\" height=\"569\" srcset=\"https:\/\/www.thenewatlantis.com\/wp-content\/uploads\/2025\/03\/Boyd-banner2-1280x569.jpg 1280w, https:\/\/www.thenewatlantis.com\/wp-content\/uploads\/2025\/03\/Boyd-banner2-1920x853.jpg 1920w, https:\/\/www.thenewatlantis.com\/wp-content\/uploads\/2025\/03\/Boyd-banner2-640x284.jpg 640w, https:\/\/www.thenewatlantis.com\/wp-content\/uploads\/2025\/03\/Boyd-banner2-1536x683.jpg 1536w, https:\/\/www.thenewatlantis.com\/wp-content\/uploads\/2025\/03\/Boyd-banner2-2048x911.jpg 2048w\" sizes=\"(max-width: 1280px) 100vw, 1280px\" \/><\/a><\/p>\n<p style=\"text-align: center;\"><strong><a href=\"\/publications\/will-ai-be-alive\">Will AI Be Alive?<\/a><br \/><\/strong><em>An essay in three parts<\/em><\/p>\n<p style=\"text-align: center;\"><a href=\"\/publications\/will-ai-be-alive\" target=\"_blank\" rel=\"noopener\">Introduction<\/a><br \/><br \/>1. Gaining Situational Awareness About the Coming Artificial General Intelligence<br \/><br \/>2. <a href=\"\/publications\/ai-will-seem-to-be-alive\">It Will Seem to Be Alive<\/a><br \/><br \/>3. <a href=\"\/publications\/we-must-steward-not-subjugate-nor-worship-AI\">We Must Steward, Not Subjugate Nor Worship It<\/a><\/p>\t<\/div>\r\n<\/div>\r\n<\/div>\n\n\n<p>Just as Smith and Baker are names derived from occupations, so is Ashburner in English and Aschenbrenner in German. The person today who perhaps best lives up to the family name is Leopold Aschenbrenner. A child prodigy who started at Columbia University at the age of fifteen and graduated in 2021 as a nineteen-year-old valedictorian, Aschenbrenner found his way to a job at OpenAI. He was fired in April 2024 \u2014 in his telling, for seeking feedback on a safety research document from outside experts, which OpenAI saw as leaking sensitive information.<\/p>\n\n\n\n<p>Aschenbrenner responded to his firing by founding an investment fund for artificial general intelligence (AGI), launched alongside the 165-page \u201cSituational Awareness: The Decade Ahead.\u201d The term \u201csituational awareness\u201d is used in military and business contexts to describe the kind of rich understanding of an environment that allows one to plan effectively and make decisions. Aschenbrenner\u2019s essay series shares his insider situational awareness of where AI is headed, with AGI \u201cstrikingly plausible\u201d by 2027 and artificial superintelligence \u2014 automated development of AI by AGI \u2014 coming hot on its heels.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"alignright size-thumbnail is-resized\"><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/www.thenewatlantis.com\/wp-content\/uploads\/2025\/03\/Aschenbrenner-portrait-640x853.jpeg\" alt=\"\" class=\"wp-image-34646\" width=\"320\" height=\"427\" srcset=\"https:\/\/www.thenewatlantis.com\/wp-content\/uploads\/2025\/03\/Aschenbrenner-portrait-640x853.jpeg 640w, https:\/\/www.thenewatlantis.com\/wp-content\/uploads\/2025\/03\/Aschenbrenner-portrait.jpeg 1080w\" sizes=\"(max-width: 320px) 100vw, 320px\" \/><figcaption class=\"wp-element-caption\">Leopold Aschenbrenner<br><a href=\"https:\/\/www.forourposterity.com\" target=\"_blank\" rel=\"noreferrer noopener\"><cite>Courtesy Leopold Aschenbrenner (fair use)<\/cite><\/a><\/figcaption><\/figure><\/div>\n\n\n<p>Silicon sand is turning into new life. Aschenbrenner writes convincingly and with grave concern that the world is not ready for the forces soon to be unleashed.<\/p>\n\n\n\n<p class=\"has-drop-cap\">It is quite possible that he is wrong. Past performance is no guarantee of future results, and what can\u2019t go on forever, won\u2019t. But equally relevant here are the maxims \u201cforewarned is forearmed\u201d and \u201cbetter safe than sorry.\u201d An unlikely but dangerous outcome may still demand our attention. This is especially the case because Aschenbrenner\u2019s argument that AGI will likely be here soon is so straightforward.<\/p>\n\n\n\n<p>It goes like this. The difference between the scant capabilities of OpenAI\u2019s GPT-2 in 2019 and the astonishing capabilities of GPT-4 in 2023 is five orders of magnitude of effective compute \u2014 a measure of a system\u2019s actual computational power, including not only the raw power of its hardware but the efficiency of its software, the level of resource overhead, and so on. On this basis, there are two things we have good reason to expect:<\/p>\n\n\n\n<ol>\n<li><strong>Another increase of five orders of magnitude in effective compute is possible on a similar timescale \u2014 that is, by 2027.<\/strong> Recall that an order of magnitude is shorthand for a tenfold increase or decrease in a given property. Even small numbers here make for large differences: If a T-Rex is twelve feet tall and a rooster is two, then the tyrant king is not even a single order of magnitude taller than the barnyard cock.<\/li>\n\n\n\n<li><strong>This increase will lead to a comparable increase in the capabilities of the AI.<\/strong> If GPT-2 could do as well on computational tasks as a preschooler, and GPT-4 can beat most high schoolers on standardized tests, then an equivalent jump to GPT-X will likely \u201ctake us to models that can outperform PhDs\u201d and experts in many fields, Aschenbrenner explains. Wharton professor <a href=\"https:\/\/x.com\/emollick\/status\/1864871107095912767\" target=\"_blank\" rel=\"noreferrer noopener\">Ethan Mollick claims that<\/a> OpenAI\u2019s o1 model, released last fall, already \u201ccan solve some PhD-level problems and has clear applications in science, finance &amp; other high value fields.\u201d<\/li>\n<\/ol>\n\n\n\n<iframe src=\"https:\/\/ourworldindata.org\/grapher\/test-scores-ai-capabilities-relative-human-performance?country=Handwriting+recognition~Speech+recognition~Image+recognition~Reading+comprehension~Language+understanding~Predictive+reasoning~Complex+reasoning~General+knowledge+tests~Math+problem-solving~Nuanced+language+interpretation~Reading+comprehension+with+unanswerable+questions~Code+generation&amp;tab=chart\" loading=\"lazy\" style=\"width: 100%; height: 600px; border: 0px none;\" allow=\"web-share; clipboard-write\"><\/iframe>\n\n\n\n<div style=\"height:25px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<p>For the effective compute of AI models to increase by another five orders of magnitude in a few years, two big obstacles would have to be overcome: more physical computing power and more efficient algorithms. Let\u2019s consider each in turn.<\/p>\n\n\n\n<p class=\"has-drop-cap\">Continuing the dramatic increase in computational ability of the last few years would require a preposterously large number of dedicated computer chips and using enormously large amounts of electricity. In 2023, Microsoft committed to buy electricity from Helion Energy\u2019s in-the-works nuclear fusion power plant, a company for which OpenAI\u2019s Sam Altman had previously helped raise $500 million; in March 2024, Amazon paid $650 million for a data center conveniently located adjacent to a nuclear power station in Pennsylvania.<\/p>\n\n\n\n<p>Those sums together are well over two orders of magnitude smaller than the $500 billion that OpenAI, Oracle, and two other firms have committed over the next four years to the new Stargate project announced by President Trump only days after he took office this year. For ease of comparison, let\u2019s say that\u2019s $100 billion per year. This is an order of magnitude less than the trillion dollars per year that Aschenbrenner expects would be needed to produce enough computing power for AGI, which would use 20&nbsp;percent or more of the total amount of electricity currently produced in the United States.<\/p>\n\n\n\n<p>This may seem laughably large, an impossible goal. We are not presently on trend to attain it, that much is clear. In 2024, Goldman Sachs released a report estimating that U.S. power demand will grow only 2.4 percent by the end of the decade, with data centers making up a large part of that demand but AI using only a fifth of the increase. Tech companies are building where the electricity is and the regulators aren\u2019t, planning data centers in Saudi Arabia and the United Arab Emirates. Aschenbrenner would prefer not to see our tech secrets handed over to America\u2019s frenemies, and he both calls for and expects a new Manhattan Project for AGI. Project Stargate, set to be centered in Texas, looks like a step in that direction.<\/p>\n\n\n\n<p>At their peaks, notes Aschenbrenner, the Manhattan and Apollo projects each cost 0.4&nbsp;percent of U.S. gross domestic product, which would today be about $100&nbsp;billion \u2014 roughly the amount of yearly investment currently committed to Stargate. Is another order of magnitude of funding possible? Probably so. Between the national security importance and the commercial possibilities, a dramatic scale-up in AI investment will likely only be forestalled by something even bigger \u2014 a government breakup of Big Tech, a Great Depression, or an act of God like a devastating solar flare, an asteroid, or an earthquake that sends California into the sea.<\/p>\n\n\n\n<p>It\u2019s also worth noting that major advances in quantum computing would make enormous investments in server farms and electricity production unnecessary. In December, Google Quantum AI announced a major breakthrough with its new Willow chip, for the first time scaling up computational power while exponentially reducing the error rate and attaining real-time error correction. And in February, Microsoft went even further with its Majorana&nbsp;1 chip, which it claims achieves an entirely new state of matter, which in turn allows for creating a new type of qubit, which is a key step toward achieving very low error rates and a practically useable quantum computer. Quantum computers are still far from ready for mass production, but they are now more probable than merely possible.<\/p>\n\n\n\n<p class=\"has-drop-cap\">The second obstacle is making algorithms more efficient. Moore\u2019s Law, describing progress in computer hardware, famously holds that the number of transistors in an integrated circuit doubles about every two years. The research firm Epoch AI has found a similar, though even faster, pattern in the progress of AI software capabilities for both image and language modeling: every eight or nine months, improvements in AI algorithms produce a doubling effect in available compute \u2014 by cutting in half the compute required. Many more computer chips and power plants will still be needed, but Aschenbrenner expects software improvements to be just as important as those in hardware. Perhaps they will be even more important, if the success of the DeepSeek-R1 model is not a fluke but rather an indication of what can be accomplished through open source on a limited computing budget.<\/p>\n\n\n\n<p>A common objection to the belief that the trend in algorithmic improvements can continue is that we are running out of unique, quality data. All of these efficient algorithms with high-powered chips have been processing increasingly large training datasets, approaching the size of the entire Internet. But as Aschenbrenner points out, large language models have been doing the equivalent of speed-reading their texts, cramming for an exam the night before and getting ready to spit out answers as quickly as possible when the test comes in the morning. This is part of why LLMs have until recently been bad at math. But better algorithms are still possible using the same datasets.<\/p>\n\n\n\n<p>Programmers are now training LLMs to process texts much more slowly, doing the algorithmic equivalent of carefully working through the practice problems in a math textbook and only then checking the answers, or of talking through a problem with a study-buddy. \u201cIn-context learning\u201d allows the LLM to carefully scrutinize a particular subset of data for answering a question.<\/p>\n\n\n\n<p>For example, the endangered language Kalamang on the island of New Guinea has fewer than 200 speakers and almost zero digital footprint, so next to nothing about it was included in Google Gemini&nbsp;1.5 Pro\u2019s training data. But when Google uploaded a reference grammar and a short dictionary to the LLM\u2019s \u201ccontext window,\u201d it was able to translate from English to Kalamang as well as human testers could when given the same resources and prompts.<\/p>\n\n\n\n<p>As of yet, there is no way for the LLM to incorporate this in-context learning into its fundamental \u201cweights\u201d or neural network architecture; there is no bridge between short-term and long-term memory. But if and when computer scientists build that bridge, it could lead to the kind of exponential gains that AlphaGo saw from playing the game Go against itself \u2014 except that these would be gains in, say, the ability to direct a drone swarm to avoid countermeasures, or the ability to manipulate humans into acting on chatbot advice.<\/p>\n\n\n\n<p class=\"has-drop-cap\">In addition to more physical computing power and more-efficient algorithms, there is a third way in which Aschenbrenner expects AI to advance by orders of magnitude in the coming years. He calls it \u201cunhobbling,\u201d recalling the binding used on horses\u2019 front legs to keep them docile.<\/p>\n\n\n\n<p>In the interest of safety and commercializability, today\u2019s LLMs are substantially hobbled in ways that can easily be undone. For the most part, they are generic chatbots given short prompts and reset after each series of queries. Unhobbling means, for example, customizing the chatbot with detailed information about the history and preferences of the person or company using it. Another example: Publicly available LLMs do not yet have access to the full tools of a computer, let alone a 3D printer. Unhobbling is when AIs are given access to both.<\/p>\n\n\n\n<p>Perhaps the most important issue is what Aschenbrenner calls the \u201ctest-time compute overhang.\u201d LLMs break down queries and responses into \u201ctokens,\u201d or fragments of vocabulary and grammar. According to OpenAI, each token covers about four characters, and is the unit for computation and thus for billing by usage. But if software and hardware increases make computation costs go down, then the \u201ccompute overhang\u201d can increase, and many more tokens can be used to answer each question.<\/p>\n\n\n\n<p>Most LLMs now sacrifice depth for efficiency, relying on what has been called \u201cSystem&nbsp;1\u201d: automatic, intuitive thinking. This is how we get results like ChatGPT insisting that there are two <em>r<\/em>\u2019s in \u201cstrawberry.\u201d But, increasingly, LLMs are being taught to pursue \u201cSystem&nbsp;2\u201d: deliberately slow, self-checking chains of reasoning. Currently, the maximum output from a public OpenAI reasoning model (o1) is 100,000 tokens. Using Aschenbrenner\u2019s assumption that a person can think quickly at about 100 tokens per minute, this is equivalent to two normal workdays of very focused work.<\/p>\n\n\n\n<p>Think of a task like \u201cread these seven academic papers, summarize them for me, come up with a new research question based on a synthesis of their findings, and outline a paper that follows up on it,\u201d for which frontier knowledge workers now use AI. But if an AI could use seven figures of tokens per answer in conjunction with the self-play and study-buddy approaches, then it could go from a stream-of-consciousness monologue to drafting, editing, and improving a response so that it represents the equivalent of not two days but a week or a month of one human worker\u2019s effort. This is the path down which Grok&nbsp;3\u2019s new \u201cthink\u201d button, Claude&nbsp;3.7 Sonnet\u2019s \u201cextended thinking mode,\u201d and similar reasoning functions have taken the first steps.<\/p>\n\n\n\n<p class=\"has-drop-cap\">Speaking of LLMs \u201cthinking\u201d and \u201creasoning\u201d is where we come to both the deepest objection and the most awful-or-awesome possibility in the trends Aschenbrenner discusses. The objection is that he has implicitly followed a common computer science practice of narrowing down what we mean by \u201cgeneral intelligence\u201d to success on metrics like AP exams, protein-folding problems, and software engineering problems.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"alignright size-thumbnail is-resized\"><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/www.thenewatlantis.com\/wp-content\/uploads\/2025\/03\/Boyd-Atlas-web-640x711.jpg\" alt=\"\" class=\"wp-image-34655\" width=\"320\" height=\"356\" srcset=\"https:\/\/www.thenewatlantis.com\/wp-content\/uploads\/2025\/03\/Boyd-Atlas-web-640x711.jpg 640w, https:\/\/www.thenewatlantis.com\/wp-content\/uploads\/2025\/03\/Boyd-Atlas-web-1280x1421.jpg 1280w, https:\/\/www.thenewatlantis.com\/wp-content\/uploads\/2025\/03\/Boyd-Atlas-web-1383x1536.jpg 1383w, https:\/\/www.thenewatlantis.com\/wp-content\/uploads\/2025\/03\/Boyd-Atlas-web.jpg 1684w\" sizes=\"(max-width: 320px) 100vw, 320px\" \/><figcaption class=\"wp-element-caption\">Boston Dynamics\u2019s Atlas<br><cite>Courtesy Boston Dynamics (fair use)<\/cite><\/figcaption><\/figure><\/div>\n\n\n<p>This is an important philosophical concern to which we will return. But the practical upshot is that as programmers focus on AI improvements in solving programming problems, and as they succeed in programming successively better artificial programmers that can program themselves, we face what has been called the \u201cintelligence explosion\u201d or, more precisely, a \u201ccapability explosion.\u201d If we grant the Machine Intelligence Research Institute\u2019s restrained definition of intelligence as the \u201cability to achieve goals in a wide range of environments,\u201d then we should expect AI programs to look decreasingly like chatbots and increasingly like agents: having agency, the ability to directly affect the real world.<\/p>\n\n\n\n<p>This will be especially clear in the development of AI-powered humanoid robots: Tesla\u2019s Optimus, Boston Dynamics\u2019s Atlas, Figure\u2019s&nbsp;02, and Sanctuary AI\u2019s Phoenix. In 2011, Marc Andreessen famously wrote that \u201csoftware is eating the world,\u201d referring to things like Amazon\u2019s victory over Borders. If Aschenbrenner is even close to right, then we haven\u2019t seen anything close to the possible impact of the digital world on the physical.<\/p>\n\n\n\n<p>Silicon Valley\u2019s old motto was \u201cmove fast and break things.\u201d Its new one may as well be \u201cmove fast and make things.\u201d<\/p>\n\n\n\n<h4 class=\"has-text-align-center\"><a href=\"\/publications\/ai-will-seem-to-be-alive\">Continue to Part 2: \u201cIt Will Seem to Be Alive\u201d  \u279e<\/a><\/h4>\n","protected":false},"excerpt":{"rendered":"<p>Part 1 of \u201cWill AI Be Alive?\u201d \u201cBurner of ashes\u201d \u2014 there\u2019s a job description one rarely sees these days. Yet going back to the Bronze Age, the ash burner had an essential if dirty job. Soaking wood ashes and water in a pot, then filtering the liquid and boiling it to evaporate all the water yields potash, useful in making dyes, soap, glass, and fertilizer. From burnt wood to fertilizer: out of the ashes comes new life. Just as Smith and Baker are names derived from occupations, so is Ashburner in English and Aschenbrenner in German. The person today&#8230;<\/p>\n","protected":false},"author":17,"featured_media":34908,"template":"","article_type":[13],"noteworthy_people":[],"topics":[2272,5025,2279],"_links":{"self":[{"href":"https:\/\/www.thenewatlantis.com\/wp-json\/wp\/v2\/article\/34696"}],"collection":[{"href":"https:\/\/www.thenewatlantis.com\/wp-json\/wp\/v2\/article"}],"about":[{"href":"https:\/\/www.thenewatlantis.com\/wp-json\/wp\/v2\/types\/article"}],"author":[{"embeddable":true,"href":"https:\/\/www.thenewatlantis.com\/wp-json\/wp\/v2\/users\/17"}],"version-history":[{"count":58,"href":"https:\/\/www.thenewatlantis.com\/wp-json\/wp\/v2\/article\/34696\/revisions"}],"predecessor-version":[{"id":35205,"href":"https:\/\/www.thenewatlantis.com\/wp-json\/wp\/v2\/article\/34696\/revisions\/35205"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.thenewatlantis.com\/wp-json\/wp\/v2\/media\/34908"}],"wp:attachment":[{"href":"https:\/\/www.thenewatlantis.com\/wp-json\/wp\/v2\/media?parent=34696"}],"wp:term":[{"taxonomy":"article_type","embeddable":true,"href":"https:\/\/www.thenewatlantis.com\/wp-json\/wp\/v2\/article_type?post=34696"},{"taxonomy":"noteworthy_people","embeddable":true,"href":"https:\/\/www.thenewatlantis.com\/wp-json\/wp\/v2\/noteworthy_people?post=34696"},{"taxonomy":"topics","embeddable":true,"href":"https:\/\/www.thenewatlantis.com\/wp-json\/wp\/v2\/topics?post=34696"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}