Continual Learning Is the Next Big Thing (Part 1)
Every era of deep learning was unlocked by removing a gradient bottleneck. The next bottleneck is that nothing a model works out survives its context window. Part 1 of 2.
Every era of deep learning was unlocked by removing a gradient bottleneck. The next bottleneck is that nothing a model works out survives its context window. Part 1 of 2.
One sum, values weighted by key-query brackets, accounts for almost every parameter in a modern network. Starting from the correspondence between a single neuron and a rank-one outer product, this post uses bra-ket notation to derive MLPs, softmax attention, effective rank, linear attention, the delta rule, Gated DeltaNet and PaTH attention as one associative memory wearing different coefficients.