An Associative Introduction to Deep Learning

if match(x, key):
    out += value

memory = {
    key1: value1,
    key2: value2,
    key3: value3,
    key4: value4,
}
for key, value in memory:
    if match(x, key):
        out += value

Neural networks are fundamentally associative. You might be tempted to view them as functions wiggling in space, but thinking of them as computing associations leads far more directly to the architectural, mechanistic, and theoretical developments of the past decade. This post won't follow the conventional ordering for learning the subject. It is meant to sit beside the standard guides and expose the structure they leave implicit. I assume some linear algebra and multivariable calculus, though the parts that carry weight get rebuilt as they arrive: if you know those tools already, the aim is to hand you another lens on them, and if you don't, most of what follows survives the translation. By the end the same structure should be visible in networks of every kind, which is what makes the more advanced material approachable.

Directions, not coordinates#

Send $x \mapsto Ax$ for any invertible $A$, every read $u \mapsto uA^{-1}$, every write $v \mapsto Av$. The two cancel wherever they meet, so the network computes what it computed before. This is all of $GL(d)$ and not just the rotations: an MLP and an attention score never measure a length or an angle. Rotations are what is left once RMSNorm and weight decay insist on a norm.

A read and a write are therefore different kinds of object. A write is a vector, an arrow you add to the stream. A read is a covector: not an arrow but a stack of level sets, and applying it to a vector counts crossings. Counting survives $A$; lengths and angles do not.

That is the case for bras and kets here, and it is not the case physics makes. The syntax closes a bra against a ket and nothing else, so $\langle u|x\rangle$ is a number and $|v\rangle\langle u|$ is an operator, while nothing contracts two bras into a number. In quantum mechanics that restriction is cosmetic, since the inner product turns any ket into a bra. A residual stream ships without that converter, so the restriction has content: the notation is a type system that refuses exactly the expressions a metric would be needed to define.

• $\langle u|x\rangle$, a read applied to the stream: fine.
• $|v\rangle\langle u|$, read then write: fine, and it is one neuron.
• $\langle u_2|v_1\rangle$, a later read against an earlier write: fine, and it is the virtual weight between two layers.
• $\langle u_1|u_2\rangle$, or a cosine between two reads, or between a read and a write: needs a metric nobody supplied.

$\phi$ is the exception. It acts on $\mathbb{R}^h$, which is not the stream but the space indexed by the neurons themselves, so it has a basis already; elementwise means something there and nothing on the stream. (RMSNorm is the other exception, and LayerNorm's mean subtraction additionally singles out the all-ones direction.)

One sum covers the rest of the post:

A query is bracketed against every key, the numbers that come back become coefficients, and the coefficients mix the values. Three choices generate almost everything below: where the keys and values come from, how much of the bracket vector each coefficient is allowed to see, and whether the memory is rebuilt from scratch at every step or carried forward and edited.

An MLP reads its keys and values out of its weights, and each $\phi_i$ sees only its own bracket. Softmax attention computes them from the sequence, and every $\phi_i$ sees all the brackets at once. Everything outside this form is the normalization gains and biases: 0.031% of GPT-2 small, 0.005% of a five-billion-parameter model, and foldable into the neighbouring matrices in any case.

1. A dictionary made of weights#

The simplest case is the one where the memory is the weights. Let an MLP with activation function $\phi$ have a hidden dimension $h$. Then:

This form reveals a simple structure: $W^\mathrm{in}$ acts as a list of keys (the lookup), $\phi$ acts as a threshold (if-else statement), and $W^\mathrm{out}$ acts as a list of values (the table). Neither list depends on the input. The layer answers every query out of the same $h$ pairs, fixed at training time, and all the input decides is which coefficients come back. How many distinct associations those $h$ pairs can hold is the next section.

A weight matrix reads two ways: its rows are bras, each producing one coefficient; its columns are kets, each scaled and summed. This distinction becomes critical when $|x\rangle$ has a representational preference. In residual architectures: If $|x\rangle$ immediately followed an activation function like ReLU or softmax, it prefers the column form (since non-linear activations make dot products meaningless). If an activation is to be applied to the result, it prefers the row form. Otherwise, it prefers the SVD form. When researchers apply linear projections, they usually have one of these forms in mind, and design architectures to apply these forms in specific, meaningful orders.

As a corollary, consider the squeeze-and-excitation layer (SE) introduced by Hu et al.. We see the same structure as the MLP, where we first apply global pooling to obtain a single vector, then split the vector into multiple distinct "heads" or convolution groups. This is actually kind of similar to mixture-of-experts without sparsity.

2. More keys than dimensions#

An informal bra-ket definition of effective rank is

so $r_{\mathrm{eff}}$ is roughly the number of key-value pairs needed for a good approximation. This makes the structural comparison clear:

• linear map: fixed coefficients, fixed key-value pairs,
• MLP: nonlinear coefficients, fixed key-value pairs,
• attention: nonlinear coefficients, token-dependent key-value pairs.

The effective rank is bounded by the number of pairs on offer. For an MLP that is at most the hidden dimension $h$; for a single attention head at one token it is at most the number of tokens attended to. In both cases the operator is a bounded sum of ket-bras in a finite-dimensional space, and the nonlinearity comes entirely from the input-dependent coefficients:

That bound is on the number of pairs. The number of features a layer can traffic in is a different question, and a much less restrictive one, because a feature is a direction rather than a coordinate. Draw $N$ unit vectors at random in $d$ dimensions and their pairwise brackets concentrate near $1/\sqrt{d}$: at $d=768$, a thousand random directions overlap by at most $0.183$ and ten thousand by at most $0.203$. Exact orthogonality caps you at $d$; near-orthogonality does not.

Retrieval is where the limit appears. Reading a linear memory $S=\sum_j |v_j\rangle\langle u_j|$ with a stored key returns

one wanted value plus a little of every other. The crosstalk grows like $\sqrt{m/d}$ while the signal stays at $1$, so the two are comparable around $m=d$; picking the nearest value is forgiving enough to survive a few multiples past that, and no further.

$\phi$ holds it off by discarding small brackets before they can mix values in. How many pairs survive depends on how sharply it does that:

A linear readout is down to half accuracy at $12d$ pairs and a ReLU at $24d$. Squaring the ReLU carries it to $768d$, and a softmax readout has not begun to fail at $2048d$, where it is holding two thousand pairs for every dimension it has. Capacity is set by the sharpness of $\phi$, not by the dimension, and the sharpest choice in common use is the exponential one, which is what softmax attention applies.

3. Everything writes to the same place#

A block does not replace the representation, it adds to it. Every dictionary in the network therefore reads from and writes into one shared space, and what travels between layers is a running sum.

Composition across layers is then immediate. A key in a later layer meets a value written by an earlier one, and all that passes between them is the number $\langle u_\ell|v_{\ell'}\rangle$: how much the later read cares about the earlier write. It is a covector against a vector, so it is a legal contraction, computable from the two weight matrices without running the model and without choosing coordinates. This is the quantity usually called a virtual weight.

It also explains why the packing above matters. Every layer writes into the same stream, so features have no choice but to coexist there as directions.

4. A dictionary the sequence writes for itself#

The MLP's entries are fixed once training ends. Attention builds its entries out of the sequence: at position $t$ the pairs $|v_i\rangle\langle k_i|$ are computed from the tokens already seen, so every position reads from a different dictionary, and that dictionary gains one entry per token. The diagram above draws exactly that: the MLP row holds the same $h$ entries everywhere while the attention row grows.

The read has the same shape as an MLP's, with one change. An MLP coefficient depends only on its own bracket. An attention coefficient is normalised against all the others:

Set beside the MLP:

Two differences: where the entries come from, and whether the coefficients are allowed to see each other. The next section removes the second.

5. When the cache collapses into one matrix#

Softmax couples the coefficients, and the coupling forces a cache. Evaluating $\operatorname{softmax}_i$ at position $t$ needs every bracket at position $t$, so every $\langle k_i|$ and every $|v_i\rangle$ has to still be there: $O(T)$ memory per token, $O(T^2)$ across the sequence.

Remove the coupling. Let each coefficient see its own bracket alone, and take the simplest such function, the identity:

A bracket is a number, so it can be written on either side of the ket. Move it across, then move the brackets:

The two lines are one string of symbols regrouped, and the step is associativity of composition. The parenthesis now depends on $t$ only through the range of the sum, so it can be carried forward instead of rebuilt:

The cache is gone. What crosses from one token to the next is a single $d_v\times d_k$ matrix with every pair ever written superposed inside it: the memory of section 1, with its entries supplied by the sequence rather than by training.

6. Forget to remember#

Superposing every pair into one matrix brings back the problem from section 2: keys that resemble each other cannot be read apart. Three characters in a sloppy piece of LLM-generated fiction, Elias (magical and brave), Elara (brave, not magical) and Silas (magical, cautious). Ask for the one who is both. Elias scores highest, but Elara and Silas score high too, so the read returns a blend. Raise the sharpness and see which readout can separate them:

Linear attention tops out near $79\%$ of the readout coming from the right entry, however hard the query pushes. Softmax keeps sharpening, because its coefficients are computed against each other and can starve the runners-up. A linear readout has to return everything it holds, in proportion.

Why no recurrence of this kind recalls perfectly. For a fixed state, $|o_t\rangle=S_t|q_t\rangle$ is a linear function of the query, and the state does not depend on the query at all. Softmax at low temperature is not linear: place $N$ keys evenly around a circle in a two-dimensional query space with $|v_i\rangle=|e_i\rangle$, and it sends the query to whichever $|e_i\rangle$ it sits nearest, a piecewise-constant map onto $N$ independent directions. No linear map does that, at any state size. The gap can only be narrowed. Two angles of attack follow.

A. Letting old entries fade#

Transformers get temporal structure from Rotary Positional Encodings, which rotate every query and key by an angle proportional to position. The bracket then depends on the gap $t-i$ rather than on $t$ and $i$ separately, and it decays as the gap grows. A head that needs to reach far back can learn to undo the rotation, which is how induction heads survive it.

Each pair of dimensions turns at its own frequency. Drag the position and watch the bracket fall off with distance:

A recurrence can do something stronger, because it can pick how fast to forget and re-pick at every token. Give each channel of the key its own decay factor:

RoPE rotates at a rate fixed per dimension pair; this shrinks at a rate learned per channel and recomputed per token. Without it, $N$ entries sharing a key drown out the $(N+1)$-th, which can contribute at most $1/N$ of the readout however important it is. With it, the old entries can be pushed down before the new one is written.

Decaying per channel looks like it fixes a preferred basis, which should be suspicious after the opening section. The restriction is only apparent: decay along any fixed set of linearly independent directions is the same model as decay along the channels, up to an invertible change of coordinates that folds into the key and query weights and leaves every bracket unchanged.

B. Erasing before writing#

Decay forgets indiscriminately. Often the model needs the opposite: a large memory it must edit at one address. Alice $\rightarrow$ sweater, Bob $\rightarrow$ hoodie, Charlie $\rightarrow$ leather jacket, and Alice changes into a coat. Everyone else's entry should survive untouched.

The state holds associations as ket-bras, and the key being overwritten is known. So read the old value out of the state with that key, $|r_t\rangle := S_{t-1}|\kappa_t\rangle$, rebuild the pair it came from as $|r_t\rangle\langle \kappa_t|$, and subtract it before writing the new one. The vector $|\kappa_t\rangle$ acts as a pseudo-query; take $\kappa_t := k_t/\lVert k_t\rVert$ for now; section C returns to what that normalisation assumes.

A full overwrite is too blunt for anything that changes gradually, a character's personality over a chapter for instance, so let the model erase and write only a fraction $\beta_t$. That is the delta rule:

Three pairs stored in a two-dimensional state. Erase at a chosen key and write a new value:

C. Why the two do not compose#

Put the two together and the recurrence is

The halves sit awkwardly. Decay is stated in a fixed basis; erasure happens along a direction that moves with the data. The obvious repair is the one that worked in section A: fold the change of coordinates $F$ into the keys and queries, and hope the delta term comes through unharmed.

It does not. Conjugation sends the projector $|\kappa_t\rangle\langle\kappa_t|$ to $F|\kappa_t\rangle\langle\kappa_t|F^{-1}$. That is still rank one and still idempotent, and it still contracts to $1$ against its own key, but it is no longer symmetric: it erases along $F|\kappa_t\rangle$ the component measured by $\langle\kappa_t|F^{-1}$, and those are two different directions. An orthogonal projection has become an oblique one, so the combined recurrence is not a reparameterisation of the fixed-basis one. The models differ.

This is the opening section's warning arriving with consequences. Writing $|\kappa_t\rangle\langle\kappa_t|$ uses one vector as both a read and a write. Nothing in a vector space lets you do that; it takes a metric to turn a ket into a bra, and dividing $k_t$ by $\lVert k_t\rVert$ is where the metric quietly enters. Channel-wise decay picks a basis, the delta rule picks an inner product, and there is no reason for the two choices to agree. That mismatch is what sent me down the rabbit hole this post came out of.

D. Every recurrence is attention with moving keys#

Unroll the recurrence, writing $A_m := \operatorname{diag}(\gamma_m)\bigl(I-\beta_m|\kappa_m\rangle\langle\kappa_m|\bigr)$ and starting from $S_0=0$:

with the factors ordered left to right in increasing $m$, since they do not commute. The query then reads

Everything between the two ends of that bracket belongs to the key side, so collect it into an effective key

and the output is a weighted sum of values, exactly as in attention:

A key is not fixed once written. It is transported forward by every operator that arrives after it, so what a token scores at position $t$ depends on the whole path from $i$ to $t$. This is the same duality that connects state-space models to attention. Note the superscript: there are $O(T^2)$ effective keys, one per pair of positions.

Two things follow. The first is that a recurrence has attention patterns after all, $\langle\tilde{k}_i^{(t)}|q_t\rangle$, which can be plotted and read the way softmax maps are; that is the subject of a later post on interpreting non-softmax attention.

The second is that nothing stops us from putting the coupling back. Run softmax over the effective scores and the result is PaTH attention:

with the values recombined exactly as in ordinary attention:

7. Where the brain writes and erases#

Nothing above was derived from neuroscience. The cerebellum, whose circuit was mapped in enough detail by the 1960s to be modelled directly, is built from these same parts.

Read the circuit from the bottom. Mossy fibres bring in the input. The granular layer expands it: about $50$ billion granule cells in a human cerebellum, more than every other neuron in the brain put together, each sampling only a handful of mossy inputs and firing sparsely. That expansion is section 2 built out of cells. A few thousand incoming fibres become an enormous, sparse, nearly decorrelated set of keys, and sparsity is what keeps the brackets between them small.

Each granule cell axon rises into the molecular layer and splits into a parallel fibre running for millimetres across the sheet, passing through the dendritic fans of hundreds of Purkinje cells on the way. One Purkinje cell sits in the path of on the order of $150{,}000$ parallel fibres, and its output is the weighted sum of them: a read of the whole key set at once, and the only output the cerebellar cortex has.

Then there is the climbing fibre. Each Purkinje cell receives exactly one, from the inferior olive, and it makes hundreds of contacts on the proximal dendrites, so a single one of its spikes is enough to seize the whole cell. When it fires, it drives long-term depression at precisely the parallel-fibre synapses that were active alongside it. A weight change proportional to presynaptic activity times an error delivered on a separate line is the delta rule, and Albus wrote it down as one in 1971.

It is the delta rule rather than plain Hebbian storage because of what the olive carries. It signals error, not target, and the deep nuclei project back onto it inhibitorily, so as a Purkinje cell learns to predict, its own output silences its teacher. The write shrinks in proportion to what is already stored. That is the term the delta rule subtracts before writing:

which is the update of section 6B rearranged, with $S_{t-1}|\kappa_t\rangle$ playing the part the nucleo-olivary pathway plays.

The hippocampus supplies the other half. CA3's recurrent collaterals, where each pyramidal cell contacts thousands of its neighbours, are a recurrent matrix storing patterns as a sum of outer products, and querying it with a fragment of a stored pattern returns the whole one. That is $S|q\rangle$ with a corrupted $|q\rangle$, and Marr described it as such in 1971, before Hopfield made the same structure famous. Upstream of it, the dentate gyrus takes cortical input and re-codes it sparsely into a far larger population, which is the same defence against collision that section 6 needed: Elias and Elara are pushed apart before anything is written, because once they are superposed no readout can separate them.

Hippocampus and neocortex store the same kind of thing at very different write strengths. The hippocampus writes at high $\beta$, changing its readout after one exposure, which is what episodic memory requires and also why its representations must be kept nearly orthogonal: a large $\beta$ on overlapping keys would overwrite the neighbours. The neocortex writes at low $\beta$ over interleaved repetitions, which is slow and needs replay to work at all, but tolerates dense overlapping codes and accumulates structure rather than episodes. The difference is $\beta$, and it is the standing account of why we have both.

These are claims about what the circuits compute, not that the brain runs gradient descent. Whether parallel-fibre LTD is sufficient for learning has been argued over for decades. The wiring is not in dispute: an expansion into sparse keys, a single reader over all of them, one line for the error.

8. Why a dictionary is enough#

One object keeps reappearing: a set of key-value pairs, a score of the query against every key, and a sum of values weighted by those scores. The MLP fixes its pairs during training, attention builds them from the sequence, and a linear recurrence collapses them into a single matrix. Only two things vary across them: where the pairs come from, and whether the scores may see one another.

Text is power-law distributed, and a memory holding a few common associations and a long tail of rare ones is the right shape for it. That is why something this plain goes so far. Where the data lacks that structure the dictionary matters less: Leela Chess Zero can cut MLP hidden width, and with it the number of stored pairs, with little effect on strength, which suggests its capacity goes into computing over the position rather than into recalling memorised moves.

@online{associative-introduction,
  author    = {Lucas Sun},
  title     = {An Associative Introduction to Deep Learning},
  year      = {2026},
  month     = {04},
  day       = {28},
  url       = {https://xtimecrystal.com/posts/260428-associative-introduction/},
}