LossFunctionMSE.png

  • a loss function ( cost, objective ) is one scalar saying how wrong the model is, which gradient descent pushes down by moving every parameter against
    • the loss is the definition of the task for a neural network; most standard ones are a negative log likelihood under some noise model
    • scale matters: if targets or features live on very different ranges, the big ones dominate the loss, so normalize first
  • MSE, regression; the gaussian noise assumption
- squaring punishes big errors hard, so it's sensitive to outliers
- mean absolute error $|y - \hat{y}|$ is the robust version, it predicts the median instead of the mean
  • cross-entropy, classification; compares the true label distribution with the predicted one
    • binary, with a sigmoid output:
    • categorical, with a Softmax output and one hot :
    • equals the entropy of the labels plus the KL from truth to prediction, so minimizing it minimizes the KL
    • rewards calibrated probabilities and punishes confident wrong answers without bound ( of something near 0 )
  • hinge, SVMs; and a raw score:
    • zero once a point is on the right side with margin 1, so only points inside or past the margin matter
  • huber, regression with outliers: quadratic for small errors, linear past