• the nonlinearity an neuron applies after its weighted sum,
    • without it a deep network collapses to one linear map: , no matter the depth
    • the name is from whether a biological neuron fires, but here it’s a smooth function, not an on/off decision
    • i think of it as what puts a fold in the input space; stacking folds is how networks carve nonlinear boundaries ( see ReLU )
  • Sigmoid , range
    • derivative , so gradients shrink every layer: the vanishing gradient problem
    • still the right output for binary classification ( a probability )
    • smooth version of the step function the original Perceptron used
  • tanh , range
    • zero centered, so nicer than sigmoid in hidden layers, but saturates the same way
  • ReLU , the default for hidden layers
    • cheap, gradient is exactly 1 for so it doesn’t vanish
    • dying ReLU: a neuron pushed to always negative input outputs 0 forever and gets no gradient
    • Leaky ReLU with small ( ~0.01 ) keeps a trickle of gradient
  • smooth ReLUs: GELU and SiLU/Swish , what modern transformers use ( SwiGLU in most LLMs )
  • Softmax
    • not elementwise like the others: turns a whole vector of logits into a distribution that sums to 1
    • output layer for multiclass, paired with cross entropy loss; also the core of Self Attention Pasted image 20240830111119.png
  • in my old XOR gate model i used ReLU for the hidden layer and sigmoid for the output
  • exploding gradients are less about the activation and more about depth/initialization; clipping and normalization ( Batch Normalization ) handle those