Introduction: Is Your Model Actually Any Good?
So, you’ve chosen your distribution, picked your link function, and let your software (like R) calculate the parameters for your Generalised Linear Model (GLM). Congratulations! But before you start using that model to predict insurance claims or mortality rates, you have to ask a crucial question: Does this model actually fit the data?
In this chapter, we look at the "diagnostic tools" of the actuarial world. Just like a doctor uses blood tests and X-rays to check a patient's health, we use residuals and statistical tests to check our model's health. We need to ensure that the patterns we see in our data are actually captured by the model and that we aren't just seeing "noise."
Note: This chapter assumes you are familiar with the basics of GLMs, such as the linear predictor and the link function, which were covered in previous chapters of the Regression section.
1. GLM Residuals: The "Leftovers"
In simple linear regression, a residual is just the difference between the observed value and the predicted value: \(y_i - \hat{y}_i\). However, in GLMs, things are a bit more complex because the variance of our data often changes depending on the mean (for example, in a Poisson distribution, the variance equals the mean).
To account for this, we use two specific types of residuals in GLMs:
A. Pearson Residuals
The Pearson residual is essentially a "standardised" version of the raw error. It divides the raw residual by the estimated standard deviation of the observation.
The formula for the \(i\)-th Pearson residual is:
\(r_i = \frac{y_i - \hat{\mu}_i}{\sqrt{V(\hat{\mu}_i)}}\)
Where:
\(y_i\) is the actual observed value.
\(\hat{\mu}_i\) is the value predicted by our model.
\(V(\hat{\mu}_i)\) is the variance function (which depends on the distribution we chose, like Poisson or Gamma).
Quick Tip: If you square all the Pearson residuals and sum them up, you get the Pearson Chi-square statistic (\(X^2\)), which we use for testing model fit!
B. Deviance Residuals
Deviance residuals are often preferred by actuaries because they relate directly to the "likelihood" of the model. They measure how much each individual observation contributes to the total deviance (the measure of "badness of fit").
While the formula is complex, the important thing to know is that they are usually more "normally distributed" than Pearson residuals, making them great for visual checks.
How do we use these residuals?
We usually plot these residuals on a graph. If our model is a "good fit," we expect to see:
1. Residuals scattered randomly around zero.
2. No obvious patterns (like a "funnel" shape or a curve).
3. No extreme outliers (points that are very far from zero).
Key Takeaway: Residuals tell us where our model is failing. If all your residuals for "young drivers" are positive, your model is consistently underestimating their risk!
2. Statistical Tests for Model Acceptability
Visual checks are great, but sometimes we need a "Yes/No" answer. We use two main statistical tests to determine if a GLM is acceptable.
A. The Likelihood-Ratio Test (LRT)
The Likelihood-Ratio Test is used when we want to compare two models: a "small" model (with fewer variables) and a "big" model (the same model but with extra variables added).
The Question: Are those extra variables actually helping, or are they just adding "noise"?
The Test Statistic:
\(LRT = 2 \times (l_{big} - l_{small})\)
Where \(l\) represents the log-likelihood of the models.
In terms of Deviance (\(D\)), this is often written as:
\(LRT = D_{small} - D_{big}\)
The Distribution: Under the null hypothesis (that the extra variables are useless), this statistic follows a Chi-square (\(\chi^2\)) distribution with degrees of freedom equal to the number of extra parameters added.
Decision Rule: If the \(LRT\) value is very large (higher than the critical value from the \(\chi^2\) tables), we reject the null hypothesis. This means the bigger model is significantly better!
B. Pearson’s Chi-square Test
This test checks the overall adequacy of the model. It asks: "Is the total difference between my data and my model small enough to be explained by random chance?"
The Test Statistic:
\(X^2 = \sum \frac{(y_i - \hat{\mu}_i)^2}{V(\hat{\mu}_i)}\)
The Comparison: We compare \(X^2\) to a \(\chi^2\) distribution with \(n - p\) degrees of freedom (where \(n\) is the number of observations and \(p\) is the number of parameters in the model).
Did you know? For a well-fitting model, the scaled deviance and the Pearson Chi-square statistic should both be approximately equal to the degrees of freedom (\(n - p\)). If they are much larger, your model might be "over-dispersed" (the data is more spread out than the model expects).
3. Interpreting Software Output (Paper B Focus)
In the CS1B exam, you will likely use R to fit a GLM. When you run a command like summary(my_model), you need to know where to look:
- Null Deviance: How well a "constant only" model fits (the worst-case scenario).
- Residual Deviance: How well your model fits. You want this to be significantly lower than the Null Deviance.
- AIC (Akaike Information Criterion): A "goldilocks" measure. It rewards models for fitting well but penalises them for having too many variables. Lower AIC is better!
- p-values for coefficients: If the p-value for a variable is \( < 0.05\), that variable is usually considered "statistically significant."
Common Mistake to Avoid: Don't just look at the p-values! A model can have significant variables but still be a "bad fit" overall. Always check the residuals and the total deviance.
Quick Review: The "Checklist" for Model Acceptability
When assessing a GLM, follow these steps:
- Check Residuals: Are they random? Are there any outliers? Use Pearson or Deviance residuals.
- Compare Deviance: Use the Likelihood-Ratio Test to see if adding a variable was worth it.
- Check Significance: Are the individual parameters (\(\beta\)) significantly different from zero?
- Global Fit: Use Pearson's Chi-square test to check if the overall model is acceptable.
- Compare AIC: If you have two different models, the one with the lower AIC is generally preferred.
Key Takeaway: No model is perfect! Our goal isn't to find a perfect model, but to find one that is useful and statistically sound. As the saying goes in statistics: "All models are wrong, but some are useful."