- recurrent neural network that works as content-addressable memory: give it a noisy or partial pattern and it falls into the closest stored one ( John Hopfield, 1982 )
- memories are attractors of the dynamics, recall is just letting the state roll downhill on an energy landscape
- example: store a few black and white letter images, flip 30% of the pixels of an “A”, run the updates, it settles back to the clean “A”
- same autoassociative memory / pattern completion story people tell about the CA3 recurrent collaterals in the Hippocampus
- setup
- binary neurons si∈{−1,+1}
- symmetric weights wij=wji, no self loops wii=0
- asynchronous update, one unit at a time: si←sgn(∑jwijsj)
- energy, basically an Ising Model spin system; each asynchronous update can only lower or keep E, so the network always ends at a fixed point ( a local minimum )
E=−21i=j∑wijsisj
- learning is one shot Hebbian learning ( “fire together, wire together” ) over patterns x1,…,xp on N neurons
wij=N1μ=1∑pxiμxjμ
- limits
- capacity is about 0.14N random patterns, past that recall collapses
- spurious states: minima that aren’t stored patterns ( mixtures of three stored ones, and every −xμ is stable too )
- the Boltzmann Machine is the stochastic version: same energy, but units flip probabilistically at a temperature, and it can have hidden units
- the energy descent was also used for combinatorial optimization like the traveling salesman problem ( Hopfield & Tank, 1985 )
- modern Hopfield networks ( continuous states, exponential energy ) store exponentially many patterns, and one update step is exactly Self Attention with the stored patterns X as keys and values and the state ξ as the query
ξnew=Xsoftmax(βX⊤ξ)
- so a [[Softmax]] attention head is a retrieval step on an associative memory, a nice bridge from 80s physics to transformers
- fed into recurrent neural networks and energy based models in general, and won Hopfield the 2024 physics Nobel ( shared with Hinton for the Boltzmann machine line )
- Continuous Attractor Networks are the cousin where memories form a continuum ( a ring or a sheet ) instead of isolated points