Commenting On Superintelligence

07/2025 - Philip

I don’t have a rigorous basis for predicting the evolution of AI systems, but then, neither does anyone else. We’re all relying on beliefs and intuitions.

Some believe confidently in the imminent arrival of super-human AI in the next few years. While I acknowledge this possibility, and I still believe there are meaningful developments occurring, here are a few points to consider:

• Historical AI forecasts have almost always been prematurely optimistic. This time could be different, but it is worth remembering.
• Compute alone is a tool. GPUs by themselves are cold metal. IQ is not a fundamental unit of physics and is an extremely noisy of a predictor. It seems odd that we need billions of dollars in GPUs and nuclear power plants to achieve human level AI when the brain runs on 20 watts. I believe there is some other quality that human beings possess, not merely characterized by FLOPS, that is responsible for human usefulness using such modest resources.
• Written text is a lossy compression of reality. Language models excel at predicting written text, but that very distribution itself removes the richness of reality. What we call insight or intuition I believe is borne out of more primordial understanding of the world that is more developed than our capacity for verbal logic.
• Fitting a model to a statistical distribution will have a hard time generalizing out of distribution. There is still something off about frontier models where I don’t feel they are improving in some general way even as various benchmark evaluations improve. They don’t work well when you discuss concepts or connections that are likely out of their training distribution, and I’ve yet to see them suggest a rich, insightful idea that isn’t already on the internet. But this should be expected when considering the objective function trained on (Kullback-Leibler divergence) minimizes the difference between the training distribution and the model distribution.
• AI research is large scale data science, not fundamental research. Very little of what is considered “AI research” has furthered some fundamental question of what intelligence is and how to create it. Rather, it is using a methodology of rapid technical improvements towards the data mining of large datasets. When we deconstruct what is going on, the basic ingredients of this engineering paradigm are no different from those of social media recommendations or quantitative trading systems: vectors to record data points from billions of humans via computer/internet use, sophisticated data servers where that data is processed and centralized, and engineers who iteratively improve the mining of that data using reliable benchmarks. It is a story of increased access to experimentation and data. In my personal view these AIs have not become anything close to a “digital Einstein”, having something close to what we call consciousness or general reasoning ability. Its powers are derived more through a form of crowdsourcing. But the developments are incredibly noteworthy as we can now share, tweak, and study a collective compression of all accessible documents, media, and human data with a few hundred gigabytes. While I suspect scaling will soon reach its limits, there are now open possibilities and lowered friction to studying this information and deploying for use in real applications.
• People overestimate the impacts of technological progress in the last few decades. Though the cultural significance of television, the internet, and smartphones are undeniable the economic stats would suggest the internet and smartphones have not made a meaningful impact on GDP or productivity. There are also a number of areas where I think things have regressed despite new technical tools (e.g. Star Wars sequels). I suspect that even with impressive systems, there will be a large stack of bottlenecks that limit the immediate economic impact.
• The journey to self-driving systems has been more arduous than it should. Driving is not a difficult human task (you press the pedal, turn the steering wheel, break when you need to, and try not to crash). Considering the resources it has taken so far, yet not fully solved, I suspect there will be many tasks that will take longer to automate than people expect. Personally, I think if government coordination weren’t an issue I would prefer to turn the US highway system into a reliable 180 MPH “zip-line” you can connect your car into. Then the problem does not require advanced computer vision but a centralized traffic management system.
• AI as a symptom of stagnation. Ignore AI progress. What else is going on in the world? In the developed world it appears rather sleepy. We have an aging population (consider the median home buyer now is almost twice it was in 1980). To echo PT, the virtual world becomes a convenient escape when it becomes difficult to affect the physical world. It is easier to launch an AI startup or move around tensors in JAX (and write this article) than it is to confront the more intimate problems of our age: crumbling infrastructure, the drug epidemic, housing affordability to name a few.

It would only be fair that I mention ways I think AI progress is not a nothing burger:

• We now have a compressed prior distribution over recorded human data that we can tractably sample from. While that distribution may not yet be able to do novel science or interact with reality autonomously, it allows us to take many plausible samples. This rapid sampling can improve the pace that humans iterate through information and ideas. Many of the current applications of LLMs and visual generation models lean on that use case (ChatGPT, MidJourney, etc.). Even if general planning or reasoning is far away, access to this format knowledge has been increased. As more people use them I would expect more ways to use the models will be discovered.
• There might be capabilities the models already know but haven’t demonstrated. There might be more general capabilities the models already know but can’t demonstrate due to the rigidness of the training distribution and loss function. Some algorithmic setup might be able to pull this general capability out somehow. I am doubtful that this can be done through LLMs alone, but I don’t discount the possibility entirely.
• Economic value can be created without full automation. If the system knows when it encounters something out of distribution and if it can accurately deal with a large portion of cases (e.g. 99%), new workflows can be applied where humans can manage a greater number of surfaces and intervene when the out-of-distribution tail events occur. Perhaps what we can achieve if not full autonomy are systems that can cover a large surface of the distribution of a task, can organize and learn from new data, are really good at determining when inputs are safely close to the data distribution, and can alert when human intervention is needed. What I am describing is not too different in structure from how most bureaucracy works (white-collar work). But the promise here is that you will now be able to handle even more work through asynchronous systems.
• A lot of resources and people have been drawn into working towards AI. David Donoho has an interesting picture for what he views as the actual irreversible singularity that has occurred: frictionless reproducibility. Rapid, open flow of information allows for faster replication and improvements to data science. Theoretical or scientific contemplation is not required to make measurable impact, thus dramatically lowering the friction to contribution. This lowering of friction leads to higher rates of investment return, leading to greater allocation of talent and money into fields that have adopted these practices–similar to what has drawn so many into computer science and software development.