Visualizing the Loss Landscape of Neural Nets: Residual vs AttnRes
The loss landscapes of 8, 12 and 16-layer nanochat models: Karpathy residuals next to attention residuals.
The loss landscapes of 8, 12 and 16-layer nanochat models: Karpathy residuals next to attention residuals.
Every era of deep learning was unlocked by removing a gradient bottleneck. The next bottleneck is that nothing a model works out survives its context window. Part 1 of 2.
One sum, values weighted by key-query brackets, accounts for almost every parameter in a modern network. Starting from the correspondence between a single neuron and a rank-one outer product, this post uses bra-ket notation to derive MLPs, softmax attention, effective rank, linear attention, the delta rule, Gated DeltaNet and PaTH attention as one associative memory wearing different coefficients.