- normalization ( feature scaling ) puts numeric features on a common scale so none wins just because its units are bigger, a standard machine learning preprocessing step
- example: house size in square feet ( ~1000s ) next to bedroom count ( 1 to 5 ); unscaled, every distance and gradient is about square feet
- for gradient descent, unequal scales stretch the loss surface into a long thin valley and steps zigzag; scaled, it’s rounder and converges faster
- min-max, to ; simple, but one outlier squashes everything else
- z-score ( standardization ), mean 0 and standard deviation 1; the usual default, but still pulled by outliers since and are
- robust scaling ( median and IQR ) when outliers are bad
- max-abs, to ; keeps sign and zeros ( good for sparse data ), outlier sensitive
- when it matters
- distance based: k-NN, K-Means, SVM; and PCA, which otherwise just finds the feature with the biggest units
- gradient trained: linear regression and Neural Networks, where large inputs saturate sigmoid-like units and destabilize training
- not for decision trees, splits don’t care about monotone rescaling
- fit the scaler on the training set only and apply that same transform to validation and test, otherwise test statistics leak in ( Network Modeling Workflow )
- keep the scaler to invert predictions back to the original units
- inside networks the same idea becomes a layer: Batch Normalization over the batch, Layer Normalization per token in transformers
- worked through for regression in Normalization, Regularization, and Gradient Descent for Linear Regression
Python
Output
