Pasted image 20240924102913.png

  • stochastic Hopfield network with hidden units, one of the first generative neural networks: same energy landscape, but units flip randomly and some are latent

    • Hopfield networks store memories as energy minima ( attractors ); a Boltzmann machine instead learns a whole probability distribution over binary patterns
    • basically an Ising Model whose couplings are learned from data ( Ackley, Hinton & Sejnowski 1985 )
      • Hinton will always remain my goat for this. founding director of UCL Gatsby btw
  • energy of a state with symmetric weights:

- the Boltzmann distribution from statistical mechanics; low energy means more probable - $Z = \sum_s e^{-E(s)/T}$, the partition function, sums over every state, which is what makes it intractable - each unit turns on with a [[Sigmoid Function|sigmoid]] of its input: $P(s_i = 1) = \sigma\left(\frac{\sum_j w_{ij} s_j + b_i}{T}\right)$ - sampling by repeatedly updating units is Gibbs sampling; slowly lowering $T$ ( simulated annealing ) helps it settle into deep minima instead of getting stuck - learning: make the data more probable than the model's own fantasies
\Delta w_{ij} \propto \langle s_i s_j \rangle_{\text{data}} - \langle s_i s_j \rangle_{\text{model}}