I'm sure you've seen the images of how standard residuals massively increase loss landscape smoothness relative to dense connections, and that this is meant to be suggestive of why they perform so much better.

A while ago Moonshot AI released the Attention Residuals paper, which gained a lot of popularity and claimed a 25% compute multiplier. The core idea is to view standard residuals as something like an SSM (without gates and fancy stuff) and swap the SSM analogy for attention. SSMs are worse than attention from a benchmark performance view, because SSMs lack the ability to perform "sharp retrieval" and gradients get washed out by noise. On the other hand, attention's softmax has so many nice properties and mathematical elegances that it just works. So instead of performing a running sum like SSMs, we perform depth-wise attention. Due to low arithmetic intensity, and also to help training, we use some more tricks like an input-independent query, RMS-normalized keys, and shared keys and values. However, the core idea is what matters.
Here is the embedding, is what sublayer reads, and is what it outputs. and are learned scalars and is a learned query.
Well, I trained two 8-layer models on 0.5B tokens using the nanochat repository; here's what it looks like for Karpathy residuals vs attention residuals. The second and third rows are the same thing for two 12-layer models trained on 1.3B tokens and two 16-layer models trained on 2.8B tokens.



As you can see, AttnRes is significantly smoother, and this is probably why it trains so much better. Wow.
(AttnRes is untuned, which is why it's so impressive that it can already match a highly tuned residual implementation (the residual design is partly from modded-nanogpt, which is really competitive), and I find it really begins to greatly outperform Karpathy's nanochat residuals when the architecture deviates from a standard dense transformer, such as when MoE is introduced or the model becomes looped/recurrent, while AttnRes needs no tuning.)
The training framework is my own fork of nanochat: nanochat-dl.
The loss landscapes are visualized using paramscape: paramscape.