stochastic Hopfield network with hidden units, one of the first generative neural networks: same energy landscape, but units flip randomly and some are latent
basically an Ising Model whose couplings are learned from data ( Ackley, Hinton & Sejnowski 1985 )
Hinton will always remain my goat for this. founding director of UCL Gatsby btw
energy of a state s∈{0,1}n with symmetric weights:
E(s)=−i<j∑wijsisj−i∑bisi,P(s)=Ze−E(s)/T
- the Boltzmann distribution from statistical mechanics; low energy means more probable
- $Z = \sum_s e^{-E(s)/T}$, the partition function, sums over every state, which is what makes it intractable
- each unit turns on with a [[Sigmoid Function|sigmoid]] of its input: $P(s_i = 1) = \sigma\left(\frac{\sum_j w_{ij} s_j + b_i}{T}\right)$
- sampling by repeatedly updating units is Gibbs sampling; slowly lowering $T$ ( simulated annealing ) helps it settle into deep minima instead of getting stuck
- learning: make the data more probable than the model's own fantasies