Regularization

Tldr

Regularization is a way to reduce overfitting by penalizing complexity and reducing variance.

$${\mathcal{L}}_{\text{reg}}(\theta )={\mathcal{L}}_{\text{data}}(\theta )+\lambda \cdot \Omega (\theta )$$

• $\lambda >0$ is the regularization strength
• $\Omega (\theta )$ is the penalty term that discourages complex models

L1 Regularization (Lasso)

$$\Omega (\theta )=\sum _{j=1}^d|{\theta }_j|=\Vert \theta {\Vert }_1$$ $${\mathcal{L}}_{\text{lasso}}(\theta )=\frac{1}{n}\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2+\lambda \sum _{j=1}^d|{\theta }_j|$$

• Adds absolute value of weights to the loss
• Encourages sparsity — many weights become exactly zero
• Performs feature selection by eliminating irrelevant features
• Not differentiable at 0 → uses subgradient methods or coordinate descent

L2 Regularization (Ridge)

$$\Omega (\theta )=\sum _{j=1}^d{\theta }_j^2=\Vert \theta {\Vert }_2^2$$ $${\mathcal{L}}_{\text{ridge}}(\theta )=\frac{1}{n}\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2+\lambda \sum _{j=1}^d{\theta }_j^2$$

• Adds squared magnitude of weights to the loss
• Tends to spread weight across features