Continual learning is the next big thing. Not linear attention, sparse attention. Not Engram, Attention Residuals, Muon, or ultra-sparse MoE. It's continual learning.
Here's why.
Deep neural networks have always been bottlenecked. The advancements of the next era always tended to have solved one of these bottlenecks. Usually it's gradients.
Back in the CNN days, the issue was scaling networks to be much deeper. Past a certain depth, adding layers made plain networks worse, even on the training set, so it would have been easy to conclude that data was inherently shallow and deep networks didn't help. The actual cause was that deep networks were chaotic at initialization, leading to noisy and useless gradients. Thus, careful model initialization and residual connections were introduced which have largely solved the issue of per-data gradients.
But now people were wondering how they could apply neural networks to natural language. Natural language posed a whole different challenge, and it seemed as though brute force scaling models was simply too inefficient for such a "human" task. However, in this case, the bottleneck was again not compute or fundamental objective but the network's gradients. The LSTM, although largely alleviating short-range chaos, turned out to still be unusable for long-range modelling: LSTM language models used about 200 tokens of context, and only the last 50 of them sharply. Its forget gates permanently erased context, killing gradient flow, but keeping them open meant chaotic dynamics as noise would accumulate. This was a true tradeoff, and what finally resolved it was removing recurrence altogether, via attention. Now attention allowed good gradient flow, the bottleneck of efficiently using within-context information was solved, without ever changing the fundamental objective.
Finally, we are here, today. Attention models have overcome their inability to directly perform recurrence via discrete Chain of Thought, which gave them incredible computational abilities while still having many decades to go before hitting diminishing returns. Transformers have largely solved within-context computation. Yet, there are already some hints of another bottleneck: context windows are limited. Some have tried to solve this via metalearning objectives, which attempt to backpropagate through a model which itself internally performs gradient descent via an inner objective. Yet, as you can imagine, that is incredibly noisy and the gradients are yet again being bottlenecked.
Actually, I want to make a quick detour to discuss how similar it is to push the frontier in machine learning as it is in physics, and really in other subjects too. When a big change is made, we are fundamentally reinterpreting what the model's purpose is. For example with ResNet, we learned that each layer does not process more abstract features directly, but rather merely "tunes" the residual representation slightly and keeps it rather anchored to a linear transformation of the input. With attention, no longer is the model modelling every single computation, but just a single transition which can be composed via CoT. Perhaps once we solve continual learning, a similar revelation will change our interpretations. Even though it's just a small correction to the code, often just a few lines, our philosophical interpretation of what's going on may be completely upended.
So why is it continual learning and not another bottleneck, like MoE or sparse attention? Well, modern models have access to incredible amounts of computation at inference. However, all of that is simply wasted. Every single mistake in a trajectory, if not RL'd, is in vain. And all the details learned through research tasks are simply forgotten by the beginning of a new chat.
Some people have identified a different but related issue. Current models require an incredible number of repetitions of data to remember incompressible facts: in one study, question-answering accuracy climbed from 25% to above 55% as the number of relevant pretraining documents went from ten to ten thousand. Even still, invertible relations like A → B may be stored but the model may still fail to flip it around (the Reversal Curse). See, I don't think humans have some magic general learning architecture which can somehow learn these zero-shot. If you told a human that Alice is Bob's wife and Charlie is Bob's brother, and asked who Charlie is to Alice, I bet you they wouldn't get it instantly either. But humans, when given time, and when using these facts to solve problems, or musing on them, can compose many simple relations along with incompressible facts to infer different variations of the same thing. Modern models can also do this via their Chain of Thought, and can certainly manipulate facts well even if only one version is in front of them: the same paper found that the curse disappears when the fact is sitting in the context. However, this is lost once the context window is cleared. And to be clear, self generated summaries do patch this up somewhat. But to store all possible combinations of a fact in a text format simply adds noise to the context. Many times as well, a mistake may occur in a situation the model already has a feature for, so correcting that mistake costs nothing extra. Yet a text summary needs to describe the specific details and context and wastes a lot of available space.
Hopefully by now you are also convinced that continual learning would solve many issues. It would allow longer chains of thought by storing key thought patterns or structure without needing a long text summary for the model to recover in full. It would solve the model's within-lifetime compute bottleneck.
That is the case for the problem, and it is where this post stops. Part 2 takes up the harder half: some possible reasons why continual learning is hard to solve, and a few approaches which may turn out to be what the original attention paper was to the transformer: the right shape, but the wrong details.