Understand how regularisation can be used to reduce overfitting in highly parameterised models

Welcome to the World of Regularisation!

Hello there! Today, we are diving into a crucial part of the CS2 Machine Learning curriculum: Regularisation. If you have ever felt like your model is working perfectly on your training data but failing miserably when you show it something new, you have encountered overfitting. Regularisation is our secret weapon to fix this.

Think of regularisation as a "speed limit" for your model. It prevents the model from getting too complicated and "memorising" the noise in your data, ensuring it remains useful for real-world predictions.

What is Overfitting and Why Does it Happen?

In highly parameterised models (models with many variables or features), the model has a lot of flexibility. It wants to pass through every single data point in the training set. While this sounds good, it's actually a problem.

The Analogy: Imagine you are studying for an exam by memorising the exact wording of past papers. If the real exam asks the same questions but changes the numbers slightly, you'll get stuck because you didn't learn the logic; you just memorised the noise. That is overfitting.

Quick Review: The Bias-Variance Trade-off
- High Variance: The model is too sensitive to small fluctuations in the training set (Overfitting).
- High Bias: The model is too simple and misses the underlying trend (Underfitting).
- Regularisation helps us find the "Sweet Spot" by slightly increasing bias to significantly reduce variance.

The Core Idea: Adding a Penalty

In standard regression (like OLS), we try to minimise the Residual Sum of Squares (RSS):
\( RSS = \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 \)

In regularisation, we don't just care about the error; we also care about how large our coefficients (\(\beta\)) are. We add a penalty term to the cost function. Our new goal is to minimise:
\( \text{Cost} = RSS + \text{Penalty} \)

By penalising large coefficients, we discourage the model from relying too heavily on any single feature or becoming too complex.

1. Ridge Regression (L2 Regularisation)

In Ridge Regression, the penalty is proportional to the square of the magnitude of the coefficients.

The Formula:
\( \text{Minimise } \sum_{i=1}^{n} (y_i - \beta_0 - \sum_{j=1}^{p} \beta_j x_{ij})^2 + \lambda \sum_{j=1}^{p} \beta_j^2 \)

Key Features of Ridge:
- It uses the L2 norm (the squared terms).
- It shrinks the coefficients toward zero, but never makes them exactly zero.
- It is great when you have many variables that all have a small effect on the outcome.

Don't worry if the math looks scary! Just remember: Ridge keeps all your variables but turns their "volume" down.

2. Lasso Regression (L1 Regularisation)

LASSO stands for Least Absolute Shrinkage and Selection Operator. Here, the penalty is proportional to the absolute value of the coefficients.

The Formula:
\( \text{Minimise } \sum_{i=1}^{n} (y_i - \beta_0 - \sum_{j=1}^{p} \beta_j x_{ij})^2 + \lambda \sum_{j=1}^{p} |\beta_j| \)

Key Features of Lasso:
- It uses the L1 norm (absolute values).
- Variable Selection: Unlike Ridge, Lasso can force some coefficients to be exactly zero.
- This effectively "deletes" useless variables from your model, leaving you with a simpler, more interpretable model.

Mnemonic Aid:
Lasso = Less variables. (It cuts out the ones you don't need!)
Ridge = Retains all variables. (It just makes them smaller.)

Key Takeaway:

Use Lasso if you suspect only a few variables are actually important. Use Ridge if you think all variables contribute a little bit to the result.

The Tuning Parameter: Lambda (\(\lambda\))

In both Ridge and Lasso, we have a symbol \(\lambda\). This is known as the tuning parameter (or complexity parameter). It controls how much we penalise the model.

How \(\lambda\) behaves:
- If \(\lambda = 0\): The penalty term vanishes. We are back to standard OLS regression (High risk of overfitting).
- If \(\lambda \to \infty\): The penalty becomes massive. All coefficients (except the intercept) will shrink toward zero (High risk of underfitting).
- The Goal: We use techniques like Cross-Validation to find the perfect value of \(\lambda\) that minimises the test error.

Did you know?
Standardising your data is essential before applying regularisation! Because we are penalising the size of \(\beta\), variables with different scales (e.g., Age in years vs. Salary in thousands) will be penalised unfairly if they aren't on the same scale.

Common Mistakes to Avoid

1. Forgetting to Scale: As mentioned above, always standardise your predictors so they have a mean of 0 and a standard deviation of 1.
2. Penalising the Intercept: We generally do not apply the penalty to the intercept (\(\beta_0\)). We want to allow the model to shift up or down freely.
3. Thinking Lasso is always better: While Lasso provides a simpler model, Ridge often performs better in terms of prediction accuracy if there are many small, active signals.

Quick Review Box

Regularisation Summary:
1. Goal: Reduce overfitting in models with many parameters.
2. Method: Add a penalty term (\(\lambda\)) to the RSS cost function.
3. Ridge (L2): Shrinks \(\beta\) towards zero. Keeps all variables. Best for many small effects.
4. Lasso (L1): Can shrink \(\beta\) exactly to zero. Performs variable selection. Best for sparse effects.
5. Selection: Use Cross-Validation to choose the best \(\lambda\).

Final Encouragement

Regularisation is one of the most practical tools in an actuary's machine learning toolkit. It moves us away from just fitting a line to a graph and toward building robust models that work on data we haven't even seen yet. Keep practicing with the formulas, and remember the "speed limit" analogy—it's all about control!