Visualizing the Loss Landscape of Neural Nets: Residual vs AttnRes
The loss landscapes of 8, 12 and 16-layer nanochat models: Karpathy residuals next to attention residuals.
I write about math, machine learning, and whatever else I find interesting.
5 posts shown
The loss landscapes of 8, 12 and 16-layer nanochat models: Karpathy residuals next to attention residuals.
Every era of deep learning was unlocked by removing a gradient bottleneck. The next bottleneck is that nothing a model works out survives its context window. Part 1 of 2.
One sum, values weighted by key-query brackets, accounts for almost every parameter in a modern network. Starting from the correspondence between a single neuron and a rank-one outer product, this post uses bra-ket notation to derive MLPs, softmax attention, effective rank, linear attention, the delta rule, Gated DeltaNet and PaTH attention as one associative memory wearing different coefficients.
This is my first post. I put off writing this for too long.
Hi! I'm Lucas Sun. Welcome to my blog.