Residual analysis and model validation

Introduction to Residual Analysis and Model Validation

Welcome to one of the most practical chapters in the Regression theory and applications section! By now, you’ve learned how to fit models like simple linear regression and Generalised Linear Models (GLMs). But how do we know if our model is actually any good?

Think of building a model like tailoring a suit. You take measurements (fit the model), but then you need to try it on and look in the mirror to see where it pinches or bags. Residual analysis is that "mirror" check. We are looking at the "leftovers" (the residuals) to see if our model missed any important patterns in the data.

In this chapter, we will learn how to use residuals to validate our models, ensuring they are reliable for making actuarial predictions.

1. What are Residuals?

A residual is simply the difference between what actually happened (the observed value) and what our model predicted would happen.

For a specific observation \(i\), the residual \(e_i\) is calculated as:

\(e_i = y_i - \hat{y}_i\)

Where:

\(y_i\) = the observed value of the response variable.
\(\hat{y}_i\) = the predicted value (or fitted value) from our model.

The Goal: If our model is perfect, the residuals should look like "white noise"—completely random with no discernible patterns. If we see a pattern, it means our model is "missing" something.

2. Residual Analysis in Linear Regression

In the standard linear regression model, we make several assumptions about our error terms (\(\epsilon\)). We use residuals to check if these assumptions actually hold true.

A. Checking for Linearity and Constant Variance

We usually plot the residuals (\(e_i\)) on the y-axis against the fitted values (\(\hat{y}_i\)) on the x-axis.

What to look for:

• A random "cloud" of points: This is good! It suggests the relationship is linear and the variance is constant (homoscedasticity).
• A "funnel" shape: If the cloud gets wider as the fitted values increase, the variance is not constant. This is called heteroscedasticity.
• A curved pattern (U-shape): This suggests the relationship between the variables isn't a straight line, and we might need a non-linear term (like \(x^2\)).

B. Checking for Normality

In CS1, we often assume that residuals follow a Normal distribution, specifically \(e_i \sim N(0, \sigma^2)\). To check this, we use a Q-Q plot (Quantile-Quantile plot).

Quick Tip: On a Q-Q plot, if the points lie roughly on a straight diagonal line, the normality assumption is satisfied. If the ends of the line curve away significantly, the data might have "fat tails" or be skewed.

C. Checking for Independence

If the data is collected over time, we plot residuals against the time order. If we see a "snake-like" pattern or a trend, the observations are likely correlated, which violates the independence assumption.

Key Takeaway: If residuals aren't random, your model is biased or inefficient. You need to go back to the drawing board!

3. Residuals in Generalised Linear Models (GLMs)

In GLMs, the "raw" residual (\(y - \hat{y}\)) is less useful because the variance often changes with the mean (e.g., in a Poisson model, the variance equals the mean). Therefore, we use "scaled" versions of residuals.

Pearson Residuals

The Pearson residual (\(r_P\)) scales the raw residual by the standard deviation of the distribution:

\(r_{P,i} = \frac{y_i - \hat{\mu}_i}{\sqrt{Var(Y_i)}}\)

This standardizes the residuals so we can compare them more easily across different values of the explanatory variables.

Deviance Residuals

The Deviance residual (\(d_i\)) measures how much each individual observation contributes to the total deviance of the model. Deviance is a measure of "discrepancy" between the model and the data.

Why use them? Deviance residuals are often preferred in GLMs because their distribution tends to be closer to a Normal distribution than Pearson residuals, making them easier to interpret in diagnostic plots.

4. Model Validation: Is the Model Acceptable?

Once we've looked at the plots, we can use formal statistical tests to decide if the model is "acceptable." The syllabus highlights two main tests for GLMs.

A. Pearson’s Chi-square Test

We can use the sum of the squares of the Pearson residuals to perform a goodness-of-fit test. The statistic is:

\(X^2 = \sum r_{P,i}^2\)

Under the null hypothesis that the model is correct, \(X^2\) follows a Chi-square (\(\chi^2\)) distribution. If the calculated statistic is much larger than the critical value from the tables, we reject the model.

B. The Likelihood-Ratio Test (LRT)

This is used to compare two models: a "Null" model (simpler) and an "Alternative" model (more complex). We want to see if adding more explanatory variables significantly improves the fit.

The test statistic is based on the deviance (\(D\)):

\(LRT = D_{Simple} - D_{Complex}\)

Rule of Thumb: If the drop in deviance (\(LRT\)) is large, the more complex model is significantly better. We compare this difference to a \(\chi^2_k\) distribution, where \(k\) is the number of extra parameters added.

5. Common Pitfalls and Tips for Students

Don't worry if this seems tricky at first! Here are some common traps to avoid:

• Mistaking "Correlation" for "Fit": A high \(R^2\) (coefficient of determination) doesn't always mean a model is good. Always check your residual plots! A model can have a high \(R^2\) but still have a non-linear pattern in the residuals.
• Outliers: One single "weird" data point can pull the regression line away from the rest of the data. Residual plots help you identify these outliers—they will be the points far away from the zero line.
• Paper B (R) Tip: In R, you will often use the command plot(model_name). This automatically generates four diagnostic plots, including the Residuals vs Fitted and the Normal Q-Q plot. Practice describing these in your own words!

Did you know? The term "residual" comes from the Latin residuum, meaning "that which remains." In actuarial science, we want what "remains" to be nothing but pure, unpredictable chance!

Summary Checklist

1. Residuals: \(y - \hat{y}\). They should look random.

2. Linear Regression Checks: Use plots to check for linearity, constant variance (homoscedasticity), and normality.

3. GLM Residuals: Know the difference between Pearson residuals (standardized by variance) and Deviance residuals (contribution to total fit).

4. Validation Tests: Use Pearson's Chi-square test and the Likelihood-ratio test to formally decide if a model is acceptable or if a more complex model is needed.