• recurrent neural network that works as content-addressable memory: give it a noisy or partial pattern and it falls into the closest stored one ( John Hopfield, 1982 )
    • memories are attractors of the dynamics, recall is just letting the state roll downhill on an energy landscape
    • example: store a few black and white letter images, flip 30% of the pixels of an “A”, run the updates, it settles back to the clean “A”
    • same autoassociative memory / pattern completion story people tell about the CA3 recurrent collaterals in the Hippocampus
  • setup
    • binary neurons
    • symmetric weights , no self loops
    • asynchronous update, one unit at a time:
  • energy, basically an Ising Model spin system; each asynchronous update can only lower or keep , so the network always ends at a fixed point ( a local minimum )
  • learning is one shot Hebbian learning ( “fire together, wire together” ) over patterns on neurons
  • limits
    • capacity is about random patterns, past that recall collapses
    • spurious states: minima that aren’t stored patterns ( mixtures of three stored ones, and every is stable too )
  • the Boltzmann Machine is the stochastic version: same energy, but units flip probabilistically at a temperature, and it can have hidden units
  • the energy descent was also used for combinatorial optimization like the traveling salesman problem ( Hopfield & Tank, 1985 )
  • modern Hopfield networks ( continuous states, exponential energy ) store exponentially many patterns, and one update step is exactly Self Attention with the stored patterns as keys and values and the state as the query
- so a [[Softmax]] attention head is a retrieval step on an associative memory, a nice bridge from 80s physics to transformers
  • fed into recurrent neural networks and energy based models in general, and won Hopfield the 2024 physics Nobel ( shared with Hinton for the Boltzmann machine line )
  • Continuous Attractor Networks are the cousin where memories form a continuum ( a ring or a sheet ) instead of isolated points