So if you haven't read the paper Situation Awareness: The Decade Ahead by Leopold Aschenbrenner you should. Everyone should. This is not to say you need to agree with every point he makes, but by reading papers like this it will make you more mentally prepared to think about the accelerating period of change that we are in. We are going through a technological shift of rapid change that people are finding difficult and are not planning for it accordingly. Because most humans are resistant to change or if they do have expectations of change it's gradual change or linear change and are not prepared for the explosion of change that is possible with the advent of AGI or artificial super intelligence (ASI). There is too much thinking that is looking at the current state and believing that upcoming changes will be gradual. And that what we have will be improvements to "chatbots" and building wrappers around GPT when the reality is that people need to start thinking about AI becoming "agentic" and the impact of "drop in" technical workers and every part of the landscape changing. So it's okay to disagree with parts of his reasoning or emphasis, but this paper does serve the purpose for helping realign mindsets.
In the paper, he is predicting AGI by 2027 and ASI by 2028-2030.
But before we get into his predictions, who is Leopold Aschenbrenner? He graduated from Columbia University at the age of 19 as valedictorian. Worked at OpenAI until being fired for "leaking" information - those back and forth allegations between them are debatable, but both sides were probably at fault or it could have been handled better. But regardless, since then he has been a prolific writer on the long term future of AI and has founded an investment company focused on AGI. Here is a very good recent interview.
So the prediction of AGI by 2027, I think the prediction of around 2027 for AGI, give or take a year, is very reasonable. Part of the issue of any prediction of AGI, is that the definition of AGI has always been pretty fuzzy. I'll save my thoughts on a discussion of the definitions of AGI for another post, but suffice it to say there is no definitive test for something being AGI, but what I expect is that around 2025-2026 what many people will be experiencing when working with these models is that they "feel" like AGI. Then there will be more improvements to the point that around 2027 a majority of reasonable people will be agreeing that AGI is here.
What I do disagree with is that after AGI arrives that almost immediately ASI models will start to show up. This is the hard takeoff scenario that many people talk about. It seems like to me there is going to have to be several breakthroughs needed before ASI, which might possibly could be in a year after AGI, but more likely much longer - maybe 10 - 20 years after AGI.
Regardless, Aschenbrenner predicts an "intelligence explosion" where AI systems rapidly evolve from human-level intelligence to vastly superhuman capabilities. He says this transition could happen extremely quickly, potentially within less than a year. If true, the implications would be staggering, suggesting a world where AI can automate AI research, compressing decades of progress into mere months.
This superintelligence could be "alien" in its architecture and training processes, potentially being incomprehensible to humans. This could lead to scenarios where humans are entirely reliant on these systems, unable to understand their operations or motivations.
He talks quite a bit about orders of magnitude "OOMs" to project progress, but my disagreement here is that technological progress of any kind is never smooth curve fitting. There are often plateaus, and sometimes these plateaus can be quite long.
He also makes one of his more astonishing economic claims of the projection of trillion-dollar compute clusters by the end of the decade. This massive industrial mobilization would involve significant increases in U.S. electricity production and the construction of large-scale GPU datacenters, which begs the question of energy as a limiting factor to all of this, which is a point Mark Zuckerberg has recently made.
As part of OpenAI's "super alignment" team he worked on ways of understanding on how the models are working internally and how they can be aligned to meet human objectives. He is of the firm belief that "super alignment" is possible. Much has been made of the abandonment in OpenAI of their alignment team with people like the brilliant Ilya Sutskever and Jan Leike leaving. I suspect that, yes they left because of perceived lack of commitment of OpenAI management to alignment and wanting more resources than what they were given, but I think the realization is that something like "super alignment" is not possible. I base this on reading some of their work and listening to them talk about what they had done for "super alignment" and it doesn't seem like much progress. And I know their group was formed with the mission that "super alignment" was a multi-year project, but still the progress doesn't seem like it was keeping pace with their expectations or the pace in general around the models. I'm not disagreeing that alignment isn't possible or guardrails needed, but some idea of "super alignment" may not be possible and that was the source of some of the conflict in OpenAI. So I think he is still overly optimistic in this paper about alignment and especially about achieving something like "super alignment."
The paper also underscores the importance of securing AI labs and their research from state-actor threats. He believe that currently leading AI labs treat security as an afterthought, leaving critical AGI secrets vulnerable. The effort to secure these secrets against espionage and other threats will be immense. Along these lines, he explores the geopolitical ramifications of superintelligence. The economic and military advantages converred by superintelligent systems could be decisive, raising concerns about an all-out race of even conflict with authoritarian regimes like China. The survival of democracy could hinge on maintaining technological preeminence and avoiding self-destruction.
He suggests that American AI labs are currently vulnerable, with inadequate security measures akin to "random startup security." The notion that Chinese intelligence could easily infiltrate these labs and exfiltrate critical AGI research is presented as a significant risk. Not to dismiss the threat of China, but he has been criticized for hyper focusing on China, when there are no shortage of governments who could be bad actors. But this is probably the best articulation I've read on the possible security threats.
He then continues this line of thought by saying that as the race to AGI intensifies, national security will become increasingly involved. By 2027/28, a government-led AGI project may emerge, as no startup can handle the complexities and risks associated with superintelligence. This scenario will likely involve unprecedented levels of mobilization and coordination within the US national security apparatus.
The paper foresees a scenario where the U.S. government takes over AGI projects, forming a "Manhattan Project" for AI by 2027/28. This would involve national security forces and intense governmental oversight to ensure the United States remains a leader in AGI development, securing AI research against espionage and potential misuse. However, I think this is really optimistic on how quickly government can move to do anything, let alone something complicated and technical like AI. I do believe the idea of AGI will become a very political issue and there will be "populist" politicians who will try and take advantage of it as a divisive issue especially to capitalize on people's fear of change - fear that will be very rational when supported by employment upheavals. But the idea that government will move at the same pace as the accelerating speed of AI seems naive.
Whatever points in this document you may agree with or disagree with, it's to be commended for its amount of detail and how thought provoking it is in considering both the extraordinary potential and the significant risks. What is important is that it challenges us to consider the rapid pace of AI development and the profound changes it could bring to society, security, and global dynamics. While some of the claims may seem far-fetched, they underscore the importance of proactive measures and robust strategies to navigate the coming period of AI advancements.
No comments:
Post a Comment