Assessing model accuracy

Exam SRM – Statistics for Risk Modeling · SOA (Society of Actuaries) · 5 min read

Welcome to Assessing Model Accuracy!

In our journey through statistical learning, we've learned that there are many different ways to build models. But here’s the big question: How do we know which model is actually the best?

Think of it like choosing a pair of running shoes. One pair might look great in the store (training data), but if they give you blisters when you actually go for a run (test data), they aren't very useful! In this chapter, we will learn how to "stress test" our models to ensure they perform well in the real world. Don't worry if this seems a bit mathematical at first—we'll break it down piece by piece.

1. Measuring the Quality of Fit

To evaluate how well a model works, we need a way to measure the distance between the actual value and the predicted value. For regression problems (where we predict numbers), the most common tool is the Mean Squared Error (MSE).

Understanding Mean Squared Error (MSE)

The MSE tells us, on average, how far off our predictions are. The formula looks like this:
\( MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{f}(x_i))^2 \)

Where:
- \( y_i \) is the actual value.
- \( \hat{f}(x_i) \) is the predicted value from our model.
- \( n \) is the number of observations.

Analogy: The Archery Target
Imagine you are shooting arrows at a target. The bullseye is the actual value (\( y_i \)). Your arrow is the prediction (\( \hat{f}(x_i) \)). The MSE measures the average "squared distance" your arrows land from the bullseye. A small MSE means you are a very accurate archer!

Training MSE vs. Test MSE

This is a crucial distinction for Exam SRM:
1. Training MSE: Calculated using the data we used to build the model. This is like practicing on a target you’ve used a hundred times.
2. Test MSE: Calculated using new data the model has never seen before. This is the "true" test of the model's accuracy.

Important Point: We don't really care how low the Training MSE is. We care about the Test MSE. Why? Because a model can "cheat" by memorizing the training data (this is called Overfitting), but it will fail miserably when it sees something new.

Quick Review:
- High Training MSE + High Test MSE = Underfitting (Model is too simple).
- Low Training MSE + High Test MSE = Overfitting (Model is too complex/memorizing noise).
- Low Training MSE + Low Test MSE = The "Goldilocks" Zone (Model is just right!).

2. The Bias-Variance Trade-off

Why does the Test MSE behave the way it does? It turns out that the expected Test MSE can be broken down into three fundamental parts. Understanding this "trade-off" is the "Holy Grail" of statistical learning.

The Three Components of Error

\( E(y_0 - \hat{f}(x_0))^2 = Var(\hat{f}(x_0)) + [Bias(\hat{f}(x_0))]^2 + Var(\epsilon) \)

1. Variance:
Variance refers to how much the model's prediction would change if we used a different training set.
Example: A very "wiggly" model that tries to touch every single data point has high variance. If you change just one data point, the whole curve changes shape.

2. Bias:
Bias is the error that comes from approximating a real-life problem (which may be very complex) with a much simpler model.
Example: If the true relationship is a curve, but you use a straight line to model it, you have high bias. You are "biased" toward a simple shape that doesn't fit reality.

3. Irreducible Error (\( Var(\epsilon) \)):
This is the "noise" in the data that we can't predict, no matter how good our model is. Think of it like the wind hitting your arrow in mid-air—it's outside your control.

Finding the Balance

As we make a model more flexible (more complex):
- Bias decreases (it fits the data better).
- Variance increases (it becomes more sensitive to specific data points).
- The Test MSE forms a U-shape. It goes down at first as bias drops, then starts climbing back up as variance takes over.

Key Takeaway: To get the lowest Test MSE, you must find the point where the increase in variance is exactly balanced by the decrease in bias.

3. Accuracy for Classification Models

So far, we’ve talked about numbers (Regression). But what if we are predicting categories (like "Default" vs. "No Default")? This is Classification.

The Training Error Rate

Instead of MSE, we use the Error Rate, which is the fraction of mistakes the model makes:
\( \frac{1}{n} \sum I(y_i \neq \hat{y}_i) \)
Simply put: (Number of wrong guesses) / (Total number of guesses).

The Bayes Classifier

The Bayes Classifier is a theoretical "perfect" classifier. It assigns each observation to the most likely class, given its features.
- The error rate produced by the Bayes Classifier is called the Bayes Error Rate.
- Did you know? The Bayes Error Rate is like the "Irreducible Error" for classification. It's the best we can possibly do, but we can't usually calculate it in real life because we don't know the true probability distribution of the data.

K-Nearest Neighbors (KNN)

Since we don't know the "Bayes" truth, we use methods like KNN to estimate it.
- KNN looks at the \( K \) observations closest to the point we want to predict and takes a "vote."
- If \( K = 1 \), the model is highly flexible (Low Bias, High Variance). This often leads to overfitting.
- If \( K = 100 \), the model is less flexible (High Bias, Low Variance). This often leads to underfitting.

Mnemonic for KNN:
Small \( K \) = Komplicated (high variance).
Large \( K \) = Konservative (low variance).

Summary of Key Concepts

Common Mistake to Avoid: Don't assume that a model with a Training MSE of zero is perfect. A zero Training MSE usually means you've overfit the model so much that it won't work on any new data!

Quick Summary:
- MSE: The main metric for regression accuracy.
- Overfitting: Getting a low training error but a high test error.
- Bias-Variance Trade-off: The struggle between a model being too simple (Bias) or too sensitive (Variance).
- Bayes Classifier: The theoretical ideal for classification.
- KNN: A flexible method where the choice of \( K \) determines the bias-variance balance.

Great job! You've mastered the basics of assessing model accuracy. Remember, the goal of statistical learning isn't to perfectly explain the past, but to accurately predict the future!

Quick check

Can you answer these now?

Open each question to check the key ideas from this chapter.

What is Mean Squared Error (MSE) and what does it measure in regression?

The MSE is defined as \( MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{f}(x_i))^2 \) and it measures the average squared difference between the actual values and the model's predictions.

What is the difference between Training MSE and Test MSE?

Training MSE is calculated using the data used to fit the model, while Test MSE is calculated using a separate set of previously unseen data to evaluate the model's predictive accuracy.

What occurs during model overfitting?

Overfitting occurs when a model yields a very low Training MSE but a high Test MSE because it has captured random noise in the training data rather than the true underlying pattern.

What are the three components of the expected Test MSE?

The expected Test MSE is the sum of the Variance of the model, the Squared Bias of the model, and the Variance of the irreducible error: \( E(y_0 - \hat{f}(x_0))^2 = Var(\hat{f}(x_0)) + [Bias(\hat{f}(x_0))]^2 + Var(\epsilon) \).

In statistical learning, how is 'Variance' defined?

Variance refers to the amount by which the estimated function \( \hat{f} \) would change if it were estimated using a different training data set.

In statistical learning, how is 'Bias' defined?

Bias is the error that results from approximating a complex real-life problem by a much simpler model.

What is Irreducible Error?

Irreducible error is the variance associated with the random error term \( \epsilon \), representing noise that cannot be reduced regardless of how well we estimate the model.

How does the Test MSE typically behave as model flexibility increases?

The Test MSE typically follows a U-shape: it decreases initially as bias drops quickly, but then increases as the variance begins to outweigh the reduction in bias.

How is the error rate calculated in classification problems?

The classification error rate is the proportion of mistakes made by the model: \( \frac{1}{n} \sum I(y_i \neq \hat{y}_i) \).

What is the Bayes Classifier and the Bayes Error Rate?

The Bayes Classifier assigns observations to the class with the highest probability given its features; the resulting error rate is the lowest possible rate, known as the Bayes Error Rate.

How does the value of K in K-Nearest Neighbors (KNN) affect model flexibility?

A small K results in high flexibility (low bias, high variance), while a large K results in low flexibility (high bias, low variance).

Ready to test yourself?

Turn these notes into exam-style practice. Get unlimited AI questions on this topic with instant marking and explanations.

Practice This Topic

More Exam SRM – Statistics for Risk Modeling chapters

* The content provided by thinka is generated by AI and may not always be accurate or up-to-date. Please use it as a supplementary resource and verify with official materials.

Put These Notes into Practice

Reading the notes is a great start. Now practise with unlimited AI-generated questions and get instant feedback. 100,000+ students are already improving their grades.

Start Practising Now View Pricing

Done reading? Test yourself with AI practice questions

Practice This Topic Now