Regularization

Sometimes the model can overfit the training data and hence underperforms with real data. To reduce complexity, we use regularization.

• Add a penalty factor on complexity i.e. number of features and values of weights.

• Lasso: Force coefficients to zero if not relevant. Ends up performing feature selection.

• Ridge: Reduces coefficients of irrelevant features but does not drop them to zero.