Regularized regression and K-nearest neighbors

Exam SRM – Statistics for Risk Modeling · SOA (Society of Actuaries) · 5 min read

Welcome to Regularized Regression and K-Nearest Neighbors!

Welcome! In this chapter, we are going to learn how to make our linear models even better. So far, you have probably learned about Ordinary Least Squares (OLS) regression, which tries to find the "best fit" by minimizing errors. But sometimes, OLS is a bit too eager to please—it can overfit the data, making it great at explaining the past but terrible at predicting the future.

We are going to explore two main ways to fix this: Regularization (which adds a "penalty" to the model to keep it simple) and K-Nearest Neighbors (a different approach that looks at "similar" data points). These tools are essential for risk modeling because they help us create models that are stable and reliable. Don't worry if this seems tricky at first; we will break it down step-by-step!

Part 1: Shrinkage Methods (Regularized Regression)

In standard regression, we want to minimize the Residual Sum of Squares (RSS). However, when we have many predictors, the model can become very complex. Shrinkage methods (also called Regularization) "shrink" the coefficient estimates (\(\beta\)) towards zero. This reduces the variance of the model at the cost of a little bit of bias.

1. Ridge Regression

Ridge regression works by adding a penalty term to the RSS. The goal is to minimize:

\(RSS + \lambda \sum_{j=1}^{p} \beta_j^2\)

Where:
- \(\lambda\) (Lambda): The tuning parameter. It controls how much we penalize the size of the coefficients.
- \(\sum \beta_j^2\): This is the L2 penalty. It squares the coefficients.

What you need to know about Ridge:
- When \(\lambda = 0\), Ridge regression is the same as OLS.
- As \(\lambda \to \infty\), the coefficients shrink toward zero (but never actually reach zero!).
- Important: Ridge regression does not perform variable selection. All variables stay in the model, just with smaller coefficients.

2. The Lasso Regression

Lasso (Least Absolute Shrinkage and Selection Operator) is very similar to Ridge, but it uses a different penalty:

\(RSS + \lambda \sum_{j=1}^{p} |\beta_j|\)

Where:
- \(\sum |\beta_j|\): This is the L1 penalty. It takes the absolute value of the coefficients.

What you need to know about Lasso:
- Unlike Ridge, Lasso can force some coefficients to be exactly zero if \(\lambda\) is large enough.
- This means Lasso performs variable selection, giving us a simpler, more interpretable model.
- Mnemonic: "Lasso Lassos" variables and kicks the useless ones out of the pen!

Quick Review: Ridge vs. Lasso
- Ridge: Good when most predictors are useful. It keeps them all but shrinks them.
- Lasso: Good when only a few predictors are actually important. It gets rid of the "noise" variables.

Summary of Regularization:

Regularization helps us solve the Bias-Variance Trade-off. By adding a little bias (shrinking the coefficients), we significantly reduce the variance, making our predictions more stable on new data.

Part 2: Choosing the Tuning Parameter (\(\lambda\))

How do we know which \(\lambda\) to use? We use Cross-Validation. Typically, we test a range of \(\lambda\) values and pick the one that results in the lowest Cross-Validation Error.

Common Mistake to Avoid:
Students often think a larger \(\lambda\) is always better because it simplifies the model. Remember: if \(\lambda\) is too large, the model becomes too simple (high bias) and misses the actual patterns in the data (underfitting).

Part 3: K-Nearest Neighbors (KNN)

Now, let's look at a completely different approach. While regression is parametric (it assumes a specific functional form like a straight line), K-Nearest Neighbors (KNN) is non-parametric. It makes no assumptions about the shape of the data.

How KNN Works:

Imagine you want to predict the price of a house. Instead of using a math formula, you look at the K houses most similar to it (its "neighbors") and take the average of their prices.

1. For Regression: Find the K closest points and calculate their average response value.
2. For Classification: Find the K closest points and take a majority vote for the class.

The Role of K:

The choice of K is critical:
- Small K (e.g., K=1): The model is very flexible and "wiggly." It has low bias but high variance (it overfits).
- Large K (e.g., K=100): The model is very smooth. It has low variance but high bias (it underfits).

Did you know?
As \(K\) increases, the flexibility of the KNN model decreases. This is the opposite of how we usually think about numbers, so keep that in mind for the exam!

The Curse of Dimensionality:

KNN sounds great, but it has a major weakness called the Curse of Dimensionality. In high-dimensional spaces (lots of predictors), the "nearest" neighbors might actually be very far away in terms of distance. This makes KNN perform much worse than linear regression when the number of predictors \(p\) is large relative to the number of observations \(n\).

Comparison: Linear Regression vs. KNN

Why choose one over the other?

Choose Linear Regression (or Ridge/Lasso) if:
- The relationship between predictors and response is close to linear.
- You have a small amount of data per predictor.
- You need to easily explain how each variable affects the result.

Choose KNN if:
- The relationship is highly non-linear or "weird."
- You have a massive amount of data and few predictors.
- Predictability is more important than understanding the "why."

Key Takeaways for Exam SRM

1. Regularization (Ridge/Lasso):
- Used to prevent overfitting and handle many predictors.
- Ridge uses \(\beta^2\) and doesn't zero out coefficients.
- Lasso uses \(|\beta|\) and can zero out coefficients (variable selection).

2. The Tuning Parameter (\(\lambda\)):
- Controls the trade-off. We find the best \(\lambda\) using cross-validation.

3. K-Nearest Neighbors (KNN):
- A non-parametric method based on similarity.
- Small K = High Flexibility / High Variance.
- Large K = Low Flexibility / High Bias.
- Suffers from the Curse of Dimensionality.

Encouragement: You've got this! Regularization and KNN are just different ways of trying to find the "sweet spot" between a model that is too simple and a model that is too complex. Practice identifying which method is best for different scenarios, and you'll be ready for any question the SRM exam throws at you!

Quick check

Can you answer these now?

Open each question to check the key ideas from this chapter.

What is the primary goal of shrinkage methods in regression?

Shrinkage methods aim to reduce model variance and prevent overfitting by penalizing the size of coefficient estimates, shrinking them toward zero.

What is the objective function for Ridge regression?

Ridge regression minimizes the sum of the RSS and an L2 penalty term: \(RSS + \lambda \sum_{j=1}^{p} \beta_j^2\).

How does the Lasso regression penalty differ from the Ridge penalty?

Lasso uses an L1 penalty, which is the sum of the absolute values of the coefficients: \(RSS + \lambda \sum_{j=1}^{p} |\beta_j|\).

When is Ridge regression generally preferred over Lasso regression?

Ridge is preferred when most predictors are useful and contribute to the response, as it keeps all variables in the model.

How is the optimal tuning parameter (\(\lambda\)) typically determined?

The optimal \(\lambda\) is determined using cross-validation to find the value that minimizes the cross-validation error.

What does it mean for K-Nearest Neighbors (KNN) to be a non-parametric method?

KNN is non-parametric because it makes no assumptions about the functional form or shape of the relationship between predictors and the response.

How does the choice of K affect the flexibility and bias-variance trade-off in KNN?

A small K increases flexibility (low bias, high variance), while a large K decreases flexibility (high bias, low variance).

What is the "Curse of Dimensionality" in K-Nearest Neighbors?

It refers to the phenomenon where, in high-dimensional spaces, the "nearest" neighbors are often very far away, significantly degrading KNN's performance.

Ready to test yourself?

Turn these notes into exam-style practice. Get unlimited AI questions on this topic with instant marking and explanations.

Practice This Topic

More Exam SRM – Statistics for Risk Modeling chapters

* The content provided by thinka is generated by AI and may not always be accurate or up-to-date. Please use it as a supplementary resource and verify with official materials.

Put These Notes into Practice

Reading the notes is a great start. Now practise with unlimited AI-generated questions and get instant feedback. 100,000+ students are already improving their grades.

Start Practising Now View Pricing

Done reading? Test yourself with AI practice questions

Practice This Topic Now