- the nonlinearity f an neuron applies after its weighted sum, a=f(w⋅x+b)
- without it a deep network collapses to one linear map: W2(W1x)=(W2W1)x, no matter the depth
- the name is from whether a biological neuron fires, but here it’s a smooth function, not an on/off decision
- i think of it as what puts a fold in the input space; stacking folds is how networks carve nonlinear boundaries ( see ReLU )
- Sigmoid σ(x)=1+e−x1, range (0,1)
- derivative σ′(x)=σ(x)(1−σ(x))≤0.25, so gradients shrink every layer: the vanishing gradient problem
- still the right output for binary classification ( a probability )
- smooth version of the step function the original Perceptron used
- tanh tanh(x)=ex+e−xex−e−x=2σ(2x)−1, range (−1,1)
- zero centered, so nicer than sigmoid in hidden layers, but saturates the same way
- ReLU max(0,x), the default for hidden layers
- cheap, gradient is exactly 1 for x>0 so it doesn’t vanish
- dying ReLU: a neuron pushed to always negative input outputs 0 forever and gets no gradient
- Leaky ReLU max(αx,x) with small α ( ~0.01 ) keeps a trickle of gradient
- smooth ReLUs: GELU xΦ(x) and SiLU/Swish xσ(x), what modern transformers use ( SwiGLU in most LLMs )
- Softmax softmax(z)i=∑jezjezi
- not elementwise like the others: turns a whole vector of logits into a distribution that sums to 1
- output layer for multiclass, paired with cross entropy loss; also the core of Self Attention

- in my old XOR gate model i used ReLU for the hidden layer and sigmoid for the output
- exploding gradients are less about the activation and more about depth/initialization; clipping and normalization ( Batch Normalization ) handle those