Hyperparameters for regularized regression

Q: What is the fundamental difference between a parameter and a hyperparameter in a GLM?

Parameters (coefficients) are learned automatically from data, while hyperparameters are set by the user before training to control the learning process.

Q: How does the hyperparameter \(\lambda\) affect a regularized regression model?

\(\lambda\) is the complexity parameter that controls the penalty strength; a higher \(\lambda\) increases the penalty, shrinking coefficients more aggressively toward zero.

Q: What occurs in a regularized regression model when \(\lambda = 0\)?

The penalty term is eliminated, and the model behaves exactly like a standard, unregularized Generalized Linear Model (GLM).

Q: What does the hyperparameter \(\alpha\) control in the glmnet package?

\(\alpha\) determines the type of penalty applied, serving as a mixing parameter between Ridge (\(\alpha = 0\)) and Lasso (\(\alpha = 1\)).

Q: What is a defining characteristic of Ridge Regression (\(\alpha = 0\))?

It uses an L2 penalty (\(\sum \beta_j^2\)) that shrinks coefficients toward zero but never forces them to exactly zero, meaning all variables are retained.

Q: Why is Lasso Regression (\(\alpha = 1\)) often used for high-dimensional data?

It uses an L1 penalty (\(\sum |\beta_j|\)) which can shrink coefficients exactly to zero, effectively performing automatic variable selection.

Q: What is Elastic Net regularization and when is it most useful?

Elastic Net (\(0 < \alpha < 1\)) combines L1 and L2 penalties and is particularly useful when there are groups of highly correlated predictors.

Q: Why is it necessary to standardize predictors before running a regularized regression?

Regularization penalties are scale-dependent; variables with larger numeric scales will be unfairly penalized more than variables with smaller scales.

Q: How does cross-validation help in selecting the value of \(\lambda\)?

It tests the model across a grid of \(\lambda\) values on different data folds to find which value results in the lowest average prediction error.

Q: When should a student choose `lambda.1se` over `lambda.min` for the final model?

Choose `lambda.1se` when the objective is a simpler, more parsimonious model that is less likely to overfit, even if it has slightly higher error than the absolute minimum.

Exam PA – Predictive Analytics · SOA (Society of Actuaries) · 5 min read

Welcome to the World of Hyperparameters!

In our previous studies of Generalized Linear Models (GLMs), we focused on finding the best "weights" (coefficients) for our variables. But what if our model is too good at its job? What if it learns the noise in our data instead of the actual signal? This is where Regularized Regression comes in, and today we’re going to learn about the "knobs and dials" we use to control it: Hyperparameters.

Don't worry if this seems tricky at first! Think of a hyperparameter like the speed limit on a road. The driver (the model) wants to go as fast as possible to reach the destination (minimize error), but the speed limit (hyperparameter) ensures the driver stays safe and doesn't crash (overfit). Let's dive in!

What are Hyperparameters?

In a standard GLM, the model calculates the coefficients (\(\beta\)) automatically. These are called parameters. However, hyperparameters are values that we, the data scientists, must choose before the model starts learning.

In the context of Exam PA and the glmnet package in R, there are two main hyperparameters you need to know:
1. \(\lambda\) (Lambda): The complexity parameter.
2. \(\alpha\) (Alpha): The elastic net mixing parameter.

Quick Review: Parameters are learned by the model from the data; Hyperparameters are set by the user to control the learning process.

1. Lambda (\(\lambda\)): The Penalty Strength

In regularized regression, we add a "penalty" to the log-likelihood function. This penalty discourages the model from making the coefficients (\(\beta\)) too large. The hyperparameter \(\lambda\) controls how severe this penalty is.

When \(\lambda = 0\): There is no penalty. The model behaves exactly like a standard GLM. This often leads to overfitting if you have too many variables.
When \(\lambda\) is very large: The penalty is massive. The model is forced to keep coefficients very small (close to zero). This can lead to underfitting.
The Goal: Find a "Goldilocks" \(\lambda\) that is just right—not too big, not too small.

Analogy: Imagine \(\lambda\) is like a "Tax" on the size of your variables. If the tax is zero, everyone spends wildly (large coefficients). If the tax is 100%, nobody spends anything (zero coefficients). We want a tax rate that keeps spending sensible!

2. Alpha (\(\alpha\)): The Type of Penalty

While \(\lambda\) tells us how much to penalize, \(\alpha\) tells us what kind of penalty to use. In Exam PA, we usually talk about three scenarios:

Ridge Regression (\(\alpha = 0\))

Ridge regression uses the L2 norm penalty, which is the sum of the squares of the coefficients: \(\sum \beta_j^2\).
Key Characteristic: It shrinks all coefficients toward zero, but they never actually reach zero. It keeps all your variables in the model, just in a "tamed" version.

Lasso Regression (\(\alpha = 1\))

Lasso regression uses the L1 norm penalty, which is the sum of the absolute values of the coefficients: \(\sum |\beta_j|\).
Key Characteristic: Lasso is "The Minimalist." It has the power to shrink coefficients exactly to zero. This means Lasso performs automatic variable selection.

Elastic Net (\(0 < \alpha < 1\))

This is a compromise between Ridge and Lasso. It’s useful when you have several variables that are correlated with each other.

Memory Aid:
Lasso = Less variables (it cuts them out).
Ridge = Retains variables (it keeps them all).

Key Takeaway: \(\alpha\) determines the style of regularization (selection vs. shrinkage), while \(\lambda\) determines the intensity.

How do we choose the best Hyperparameters?

We don't just guess! We use a process called Cross-Validation (CV). Specifically, the R function `cv.glmnet` is your best friend on the exam.

Step-by-Step Selection:

1. Standardize the Data: Before regularizing, always ensure your numeric predictors are on the same scale. (R's `glmnet` does this by default!). If you don't, a variable with a large scale (like "Annual Income") will be penalized differently than a small scale (like "Age").
2. Run Cross-Validation: The computer tries a whole range of \(\lambda\) values.
3. Pick the Winner: We usually look for one of two \(\lambda\) values provided in the output:

lambda.min: The value of \(\lambda\) that gives the lowest cross-validation error. Use this if you want the most accurate model.
lambda.1se: The largest \(\lambda\) such that the error is within one standard error of the minimum. Use this if you want a simpler, more parsimonious model (especially for Lasso).

Did you know? In Exam PA, if the prompt asks for a "simple" or "interpretable" model, lambda.1se is often the preferred choice because it results in fewer non-zero coefficients!

Common Mistakes to Avoid

1. Forgetting to Scale: As mentioned, regularization is scale-dependent. Luckily, `glmnet` handles this, but you should mention it in your written report to show the graders you understand the concept!

2. Confusing \(\alpha\) and \(\lambda\): Remember that \(\alpha\) is usually a choice you make (or test via a loop), while \(\lambda\) is the specific value found via the cross-validation "elbow" or "dip" in the error plot.

3. Overlooking Categorical Variables: When using Lasso, a categorical variable with 5 levels might have 3 levels "zeroed out" and 2 levels remain. This is how Lasso handles grouping.

Summary Checklist

- Regularization prevents overfitting by penalizing large coefficients.
- \(\lambda\) (Lambda) controls the penalty's strength. High \(\lambda = \) simpler model.
- \(\alpha\) (Alpha) controls the penalty type. \(\alpha=1\) is Lasso (selection); \(\alpha=0\) is Ridge (shrinkage).
- Cross-Validation is the tool used to find the optimal \(\lambda\).
- lambda.min is for best fit; lambda.1se is for a simpler, more robust model.

Great job! You've mastered the dials of regularized regression. These tools are essential for building GLMs that perform well on new, unseen data—which is exactly what an actuary needs to do!

Quick check

Can you answer these now?

Open each question to check the key ideas from this chapter.

What is the fundamental difference between a parameter and a hyperparameter in a GLM?

Parameters (coefficients) are learned automatically from data, while hyperparameters are set by the user before training to control the learning process.

How does the hyperparameter \(\lambda\) affect a regularized regression model?

\(\lambda\) is the complexity parameter that controls the penalty strength; a higher \(\lambda\) increases the penalty, shrinking coefficients more aggressively toward zero.

What occurs in a regularized regression model when \(\lambda = 0\)?

The penalty term is eliminated, and the model behaves exactly like a standard, unregularized Generalized Linear Model (GLM).

What does the hyperparameter \(\alpha\) control in the glmnet package?

\(\alpha\) determines the type of penalty applied, serving as a mixing parameter between Ridge (\(\alpha = 0\)) and Lasso (\(\alpha = 1\)).

What is a defining characteristic of Ridge Regression (\(\alpha = 0\))?

It uses an L2 penalty (\(\sum \beta_j^2\)) that shrinks coefficients toward zero but never forces them to exactly zero, meaning all variables are retained.

Why is Lasso Regression (\(\alpha = 1\)) often used for high-dimensional data?

It uses an L1 penalty (\(\sum |\beta_j|\)) which can shrink coefficients exactly to zero, effectively performing automatic variable selection.

What is Elastic Net regularization and when is it most useful?

Elastic Net (\(0 < \alpha < 1\)) combines L1 and L2 penalties and is particularly useful when there are groups of highly correlated predictors.

Why is it necessary to standardize predictors before running a regularized regression?

Regularization penalties are scale-dependent; variables with larger numeric scales will be unfairly penalized more than variables with smaller scales.

How does cross-validation help in selecting the value of \(\lambda\)?

It tests the model across a grid of \(\lambda\) values on different data folds to find which value results in the lowest average prediction error.

When should a student choose `lambda.1se` over `lambda.min` for the final model?

Choose `lambda.1se` when the objective is a simpler, more parsimonious model that is less likely to overfit, even if it has slightly higher error than the absolute minimum.

Ready to test yourself?

Turn these notes into exam-style practice. Get unlimited AI questions on this topic with instant marking and explanations.

Practice This Topic

More Exam PA – Predictive Analytics chapters

* The content provided by thinka is generated by AI and may not always be accurate or up-to-date. Please use it as a supplementary resource and verify with official materials.

Put These Notes into Practice

Reading the notes is a great start. Now practise with unlimited AI-generated questions and get instant feedback. 100,000+ students are already improving their grades.

Start Practising Now View Pricing

Done reading? Test yourself with AI practice questions

Practice This Topic Now