Pasted image 20240808134117.png

  • the unit of a neural network: weight the inputs, add a bias, squash

- **weights** $w_i$ are the learned [[Parameter|parameters]] saying how much each input matters - **bias** $b$ shifts the threshold, the intercept of the [[Linear Equations|linear equation]] - $f$ is the [[Activation Functions|activation function]]; drop it and stacked neurons are just [[Linear and Nonlinear Regression|linear regression]] - with $f$ the [[Heaviside Function|step function]] this is exactly the [[Perceptron]] - example: inputs $x = (1, 2)$, weights $w = (0.5, -1)$, $b = 1$ - $z = 0.5 - 2 + 1 = -0.5$, so [[ReLU]] gives $0$ and [[Sigmoid Function|sigmoid]] gives $\approx 0.38$ - a layer is many of these in parallel, $a = f(Wx + b)$, so the whole thing is matrix multiplies plus elementwise nonlinearities - loosely modeled on the biological [[Neuron|neuron]]: [[Notes/Computational Neuroscience/Brain/Dendrites|dendrites]] in, soma sums, axon out - the analogy is thin; a real cortical neuron does nonlinear dendritic computation and needs a several layer deep net to imitate ( [[Dendrites Why Biological Neurons Are Deep Neural Networks|video]], [[Pyramidal Neurons]] ) - and it communicates in spikes over time, which [[Spiking Neural Network|spiking networks]] keep