How Bad Would RSI Be Exactly?
Run in circles, scream and shout! ...Or not, actually. Or just chill. No, really.
Recursive self-improvement. A term to send chills down every human spine. AI that is so smart it can improve itself, autonomously and relentlessly, better and better faster and faster, until the improvements snowball into an exponential curve that renders all of history obsolete and utterly transforms -- or ends -- the human experience. Right?
...maybe not?
We are absolutely in the foothills of RSI right now. I’ve been running a whole lot of (non-LLM) AI experiments myself over the last couple of weeks, and there is zero doubt in my mind that frontier LLMs have greatly increased the rate at which I can run them, and better ones will increase the rate further. Gulp. Panic! Sackcloth-and-ashes time! The curve is about to bend!
...unless, that is, it isn’t.
Four years ago I wrote a take on AI doom which included a puzzled prologue:
Let’s look at the problem with at least a little more rigor ... Before writing this I had assumed other people had crafted multiple Drake Equations For AI Risk, but, to my surprise, found none. In their absence, I give you my own.
The same thing seems to be the case for RSI? He said, genuinely baffled? Don’t get me wrong, there’s a lot of good previous work, but ... while everyone just knows RSI would be earth-shaking ... no one seems to have actually modeled its possible permutations. And when I do, I get some surprising results.
Some caveats: lots of people are way more qualified than me to do this analysis; all I have is a dusty undergrad engineering degree. (Indeed I’m writing this in part in the hopes more-qualified people will disagree, but now more rigorously.) That said, the central thesis -- that in most scenarios RSI is actually not that big a deal! -- should read as pretty obvious and intuitive if you have, like, strong high-school math.
The State Of The Art
In Nick Bostrom’s Superintelligence, a fascinating book and significant influence on some of my own (fictional…) work, he models the speed of AI progress as:
or
where h is our vastly oversimplified one-dimensional measure of intelligence / capability, O(h) is the effective effort being applied to improving it, and R is recalcitrance, or how difficult it is to increase intelligence.
The fear of RSI is that, basically, it will lead to exponential growth in intelligence, also known colorfully as FOOM.
The arguments against a smooth exponential “intelligence explosion” are generally that it’s not realistic in the real world: hardware limitations, experimental wall-clock time, AI’s famous ‘jagged frontier,’ etc., will interrupt O(h) no matter how smooth it may seem on paper. These arguments are probably correct! However here I am disregarding them and stipulating, for the sake of argument, that all will be mere ephemeral speed bumps in the way of runaway O(h).
Artificial Intelligence, Natural Recalcitrance
Way back in 2015, Sebastian Benthall pointed out:
If recalcitrance is constant ... [h] will be exponentially increasing in time t. This is the “intelligence explosion” that gives Bostrom’s argument so much momentum. However, it’s important to remember that recalcitrance may also be a function of intelligence. Bostrom does not mention the possibility of recalcitrance being increasing in intelligence.
This is kind of a big deal! It means the actual equation we’re dealing with is
RSI can lead to an O(h) that is as exponential-growth as all-get-out, but if R(h) is even slightly larger, AI progress will soon hit a wintry wall. As we all learned in grade school, if you only worry about the numerator, without considering the denominator, you’re gonna get a pretty bad grade.
Three Scenarios for RSI
Now, when people say “RSI is exponential,” they aren’t wrong … but there are several very different meanings of “exponential.” In fact there are (at least) three different mathematical regimes for RSI, and which one applies really matters a lot.
1. Exponential growth from linear RSI
Here, the smarter a system gets, the speed at which it improves itself is in linear proportion to its current capability, such that:
With constant recalcitrance, we get:
and therefore
This is ‘ordinary’ exponential growth over time, as depicted above. Pretty FOOMy, but far from the most dramatic form of FOOM.
2. Polynomial RSI
Now suppose optimization power scales as the square, or cube, or Nth power, of h:
Then for constant recalcitrance we get:
which is FOOMier yet.
3. Exponential RSI
Now suppose that the smarter a system gets, each capability increment multiplicatively increases its improvement power.
This is really very super FOOMy indeed with constant recalcitrance.
It may seem that #3 is both a very strong assumption to make about RSI, and also kind of a pointless and unnecessary one. On the contrary: as we will see below, this is a very important distinction! In fact it seems likely this is the only RSI scenario which even might lead to a FOOMy intelligence explosion. Because…
Three Scenarios for Recalcitrance
1. Gaussian
Suppose that at very low (h), intelligence is a curve that is hard to start climbing at all: you need memory, learning, generalization, world-modeling, feedback, search, whatever the minimum machinery is. Then there’s a middle regime where intelligence becomes easier to improve: once the machinery exists, whether via evolutionary or engineering means, there are gradients to descend, tools to use, experiments to run. But at some point, the paths thin out again, and each additional (h) increment requires rarer and rarer combinations of architecture / data / energy / insight.
This sort of matches our intuition? Evolution found it pretty hard to evolve our intelligence; we’re so much smarter than other animals that of all the species only we show anything remotely close to enough intelligence to build something that can achieve RSI (stipulated there are many very plausible reasons for this other than recalcitrance!); but we’d generally intuitively agree that increasing intelligence is still a hard thing to do.
As a verrrrrry simple model -- don’t worry I’m not really going to run with it -- let ‘accessibility’, the inverse of recalcitrance, be a famous “bell curve,” ie a Gaussian, around some easiest-to-reach capability level (h*):
Then inverse rarity in the curve’s right tail, used as a crude proxy for cost, is:
and marginal recalcitrance is roughly
This is extremely anti-intelligence-explosion. Like, so not gonna have to worry about it.
Now, I strongly doubt this is the real R(h) (see below.) But this does illustrate that a familiar, if super crude, bell-curve intuition about cognitive capability implies the opposite of an intelligence explosion, even under the most powerful RSI regime above!
I should probably point out that exponential curves are famously weak on their left-hand side before bending to become extremely strong, so “but LLMs have gotten so much better so fast” is not a counterargument here. O(h) and R(h) are functions whose ratio may vary a lot along their x-axis, and also we have recently poured many orders of magnitude more effort into LLM progress.
2. Talebian
However. As the great-if-curmudgeonly Nassim Taleb points out -- it’s kind of his whole thing -- expecting curves to be “thin-tailed” Gaussian when they are actually “fat-tailed” is the cause of whole categories of human catastrophes.
Suppose instead the right tail of intelligence is a power law:
Then, if we use inverse rarity as a proxy for cost, we get
and marginal recalcitrance is only polynomial
This is a very different world. If RSI is linear, scenario 1 above, then again we need not fear and dare not hope for an intelligence explosion; not gonna happen. But in the case where O(h) is an exponential function itself ... well, over time, an exponential always leaves a polynomial in the dust. And in the case where both are polynomial? Then it all depends on which polynomials exactly -- it’s a dance on a knife-edge.
3. Chinchillian
However. We actually have empirical data on not one but two plausible recalcitrance curves! Sort of. Not data on recalcitrance directly, but data on modern LLM scaling laws and Hutter Prize compression difficulty.
The Chinchilla scaling laws state, basically, oversimplifying, that excess loss decreases as a power law in compute as model size and data scale. Let’s define h in this regime as reduction of error, such that reducing error rates from 50% to 25%, and reducing them from 1% to 0.5%, both improve h by the same increment, so
That may sound handwavey, but in general this maps well to predictive / compressive / loss-function capability -- equal multiplicative reductions in uncertainty correspond to roughly equal advances in predictive capability -- and is also reasonably consistent with h as rarity in the search space in the Talebian and Gaussian regimes above.
Then Chinchilla-like recalcitrance is polynomial in loss but exponential in log-loss, and looks like
Similarly the Hutter Prize shows slow improvements to data compression over time, given fixed constraints, pointing to a classic diminishing-returns curve. It’s broadly accepted that compression is a good proxy for intelligence … and while LLMs can do better than the Hutter state-of-the-art, note the fixed constraints; they just push the additional, disproportionate costs into training, which reinforces the recalcitrance.
We definitely cannot draw any dispositive conclusions from what we know today. Having said that it sure looks like a Chinchillian regime from here. In a compute-constrained regime, scaling compute is likely a good first-order proxy for the effort required to improve capability, and the Hutter Prize data points the same way.
Note that this scenario, exponential recalcitrance, will always swamp linear or polynomial RSI -- scenarios 1 and 2 above -- over time. But with scenario 3, where O(h) is also exponential, we get a total growth rate of
once again a knife-edge; growth rate over time depends entirely on whether b, the optimization power is bigger than a, the recalcitrance power! Note however
again, this is only relevant if RSI is not just exponential growth but also exponential optimization, do not confuse the two
b and a probably aren’t constants across h ... so even in this regime, as with the polynomial-polynomial one, we would likely get periods of exponential growth, periods of near-flatlining, and everything in between.
4. Kaleidoscopian
However. There is absolutely no reason to expect a single closed-form function to describe either O(h) or R(h). In practice intelligence is almost certainly multivariate and both O and R will be superpositions of many functions, some of which will probably be discontinuous in h. But the above discussion and toy models are still useful for discussion purposes since at any point on h the marginal gain and recalcitrance are probably best described by single functions.
The Table
Conclusions
So let me once again stipulate that my math is super rusty and I look forward to better-qualified people fact-checking and/or correcting me here. That said, there are two conclusions here:
Confidently: the effect of RSI is highly dependent on both what level of RSI and the recalcitrance regime, and as shown in the table above, there are many permutations of these in which the introduction of RSI is anticlimactic rather than transformative, overwhelmed by increasing recalcitrance as capabilities increase. This really just falls out of the math.
Much less confidently: the available evidence indicates exponential recalcitrance, a denominator which will strongly undercut RSI’s numerator.
It is worth noting that even if we do have e.g. exponential recalcitrance and polynomial RSI, the former will eventually win out, but the introduction of RSI is likely to lead to a short-term burst of capability growth before being swallowed up by recalcitrance. But it might not, or the burst might be very brief! Short-term effects depend not on the functions’ shapes, but on their exact parameters and where we are on the curve … and on all the hardware / experimental-time / jagged-frontier objections I’ve handwaved away throughout this post. So:
RSI seems really quite unlikely to lead to a FOOMy intelligence explosion, and
by itself its introduction doesn’t tell us what will happen in the short term either.
Much remains uncertain! But for now my tl;dr is we’re most likely in a world in which, baby, don’t fear the RSI. (Or, if you’re an AI optimist, sorry baby, don’t hope for The Culture anytime soon.)