Lucas Sun

Visualizing the Loss Landscape of Neural Nets: Residual vs AttnRes

I'm sure you've seen the images of how standard residuals massively increase loss landscape smoothness relative to dense connections, and that this is meant to be suggestive of why they perform so much better.

The loss surfaces of ResNet-56 without and with skip connections, from the original paper.
The with/without residual connections plot from the original paper

A while ago Moonshot AI released the Attention Residuals paper, which gained a lot of popularity and claimed a 25% compute multiplier. The core idea is to view standard residuals as something like an SSM (without gates and fancy stuff) and swap the SSM analogy for attention. SSMs are worse than attention from a benchmark performance view, because SSMs lack the ability to perform "sharp retrieval" and gradients get washed out by noise. On the other hand, attention's softmax has so many nice properties and mathematical elegances that it just works. So instead of performing a running sum like SSMs, we perform depth-wise attention. Due to low arithmetic intensity, and also to help training, we use some more tricks like an input-independent query, RMS-normalized keys, and shared keys and values. However, the core idea is what matters.

Standard:hk=hk−1+vk−1=∑i<kviKarpathy:hk=λk (hk−1+vk−1)+μk v0Attention:hk=∑i<kαk,i vi,αk,i=softmax⁡i<k ⁣(uk⋅virms⁡(vi))\begin{aligned} \text{Standard:}\quad & h_k = h_{k-1} + v_{k-1} = \sum_{i<k} v_i \\[6pt] \text{Karpathy:}\quad & h_k = \lambda_k\,(h_{k-1} + v_{k-1}) + \mu_k\, v_0 \\[6pt] \text{Attention:}\quad & h_k = \sum_{i<k} \alpha_{k,i}\, v_i, \qquad \alpha_{k,i} = \operatorname*{softmax}_{i<k}\!\left(\frac{u_k \cdot v_i}{\operatorname{rms}(v_i)}\right) \end{aligned}

Here v0v_0 is the embedding, hkh_k is what sublayer kk reads, and vk=fk(norm⁡(hk))v_k = f_k(\operatorname{norm}(h_k)) is what it outputs. λk\lambda_k and μk\mu_k are learned scalars and uku_k is a learned query.

Karpathy residualshk = Σi<k viOne shared stream: every sublayer reads the running sum and adds to it.embedding123456Attention residualshk = Σi<k αk,i vi , αk = softmax(uk · vi)No stream: every sublayer reads its own softmax-weighted mix of all earlier outputs.ΣΣΣΣΣΣembedding123456
How information moves between sublayers. Karpathy residuals are nanochat's default: a residual stream (with a few learned scalars). Attention residuals give every sublayer its own softmax-weighted average of the embedding and every earlier output.

Well, I trained two 8-layer models on 0.5B tokens using the nanochat repository; here's what it looks like for Karpathy residuals vs attention residuals. The second and third rows are the same thing for two 12-layer models trained on 1.3B tokens and two 16-layer models trained on 2.8B tokens.

Six maps of the loss around a trained model along two random filter-normalized directions: Karpathy residuals and attention residuals, at 8, 12 and 16 layers. The attention residual basins are rounder and wider.
Loss (nats per token) above each trained model, along two random filter-normalized directions. The dot is the trained model; darker is closer to its loss.
The loss along each of the two directions through the trained model, Karpathy residuals and attention residuals on the same axes, at 8, 12 and 16 layers.
The same landscapes cut along each direction through the trained model (dashed line). A symmetric basin mirrors across that line.
Validation loss during training for the 8, 12 and 16-layer pairs, Karpathy residuals and attention residuals on the same axes. The two curves nearly coincide at every depth.
Validation loss during training (lower is better).

As you can see, AttnRes is significantly smoother, and this is probably why it trains so much better. Wow.

(AttnRes is untuned, which is why it's so impressive that it can already match a highly tuned residual implementation (the residual design is partly from modded-nanogpt, which is really competitive), and I find it really begins to greatly outperform Karpathy's nanochat residuals when the architecture deviates from a standard dense transformer, such as when MoE is introduced or the model becomes looped/recurrent, while AttnRes needs no tuning.)

The training framework is my own fork of nanochat: nanochat-dl.

The loss landscapes are visualized using paramscape: paramscape.