Welcome to Model Selection!
Hi there! If you’ve made it this far in your ASTAM journey, you already know how to estimate parameters for different models. But here is the big question: Which model should you actually use?
In this chapter, we are going to learn how to play "matchmaker" between our data and our mathematical models. We want a model that fits well but isn't unnecessarily complicated. We will explore three main ways to decide: looking at pictures (Graphical Procedures), running formal "quality checks" (Hypothesis Tests), and using mathematical "penalties" for complexity (Score-based Criteria).
Don’t worry if this seems a bit abstract right now. Think of it like buying a pair of shoes—you want them to look good (graphs), pass a comfort test (hypothesis tests), and not be overpriced for what they offer (score-based criteria). Let’s dive in!
1. Graphical Procedures: The Eye Test
Before we crunch heavy numbers, we usually start by just looking at the data. Graphical procedures help us see where a model fits well and where it fails.
The Empirical vs. Fitted Distribution
We compare the Empirical Distribution Function \(F_n(x)\)—which is what the data actually did—against the Fitted Distribution Function \(F^*(x)\)—which is what our model says should happen.
The Difference Plot
A difference plot graphs the value of \(D(x) = F_n(x) - F^*(x)\).
• If the graph stays close to zero, the model is doing a great job.
• If there are big "bumps" or "dips," the model is failing in those specific areas (like the tails or the middle).
p-p Plots (Probability-Probability Plots)
In a p-p plot, we plot the pairs \((F^*(x_j), F_n(x_j))\).
• If the model is perfect, all points will fall on a 45-degree straight line from (0,0) to (1,1).
• Quick Tip: p-p plots are excellent for seeing how well a model fits the "middle" of the data, but they often mask problems in the tails (the very large or very small values).
Key Takeaway
Visual checks are great for a "vibe check" of the model, but they are subjective. One person might think a curve looks "close enough," while another might disagree. That’s why we need the next section: Hypothesis Tests!
2. Hypothesis Tests: The Formal Decision
These tests give us a mathematical "Yes" or "No" (or rather, "Fail to Reject" or "Reject") regarding whether a model is a good fit.
The Kolmogorov-Smirnov (K-S) Test
This is one of the most famous tests in actuarial science. It looks for the single biggest gap between our data and our model.
The Formula:
\(D = \max |F_n(x) - F^*(x)|\)
• How it works: We calculate the distance between the model and the data at every point. The "Test Statistic" is simply the largest distance we find.
• Limitation: The K-S test is strictly for individual (not grouped) data and works best when the parameters are not estimated from the data (though we often use it anyway in practice with adjustments).
The Anderson-Darling (A-D) Test
Think of the A-D test as the "Strict Teacher" version of the K-S test. While the K-S test treats all errors the same, the A-D test puts more weight on the tails.
Why does this matter? In insurance, we care a lot about the "tails" (the massive, rare claims). A model that fits the average claim perfectly but misses the huge claims is dangerous! The A-D test helps catch this.
The Likelihood Ratio Test (LRT)
This is a very common exam topic. We use this when we have two nested models.
Analogy: Think of a "Basic Model" (e.g., Exponential) and an "Advanced Model" (e.g., Gamma). Since the Exponential is just a Gamma with one parameter fixed, they are nested.
The Process:
1. Calculate the Log-likelihood of the simpler (restricted) model: \(\ell_0\).
2. Calculate the Log-likelihood of the complex (unrestricted) model: \(\ell_1\).
3. Compute the statistic: \(T = 2(\ell_1 - \ell_0)\).
4. Compare \(T\) to a Chi-square (\(\chi^2\)) distribution.
Degrees of Freedom: The degrees of freedom for the \(\chi^2\) test is the difference in the number of parameters between the two models.
Key Takeaway
Hypothesis tests provide a structured way to reject bad models. Remember: If the p-value is small (usually < 0.05), the model is a bad fit!
3. Score-Based Criteria: Balancing Fit and Complexity
If you keep adding parameters to a model, the fit will always get "better," but the model becomes a mess to use. This is called overfitting. Score-based criteria "punish" the model for having too many parameters.
Akaike Information Criterion (AIC)
AIC attempts to find the model that explains the most data with the fewest parameters.
The Formula:
\(AIC = -2\ell + 2k\)
(Where \(\ell\) is the log-likelihood and \(k\) is the number of parameters).
Schwarz Bayesian Criterion (SBC or BIC)
SBC is similar to AIC but has a "heavier" penalty for adding parameters, especially when you have a lot of data points (\(n\)).
The Formula:
\(SBC = -2\ell + k \ln(n)\)
(Where \(n\) is the number of observations).
Which one do I choose?
For both AIC and SBC, the smaller the value, the better the model.
• Did you know? Because \(\ln(n)\) is usually greater than 2, SBC is almost always stricter than AIC. It prefers simpler models more than AIC does.
Common Mistakes to Avoid
• Mixing up AIC/SBC: Students often think a "high score" is good. Nope! Think of it like golf—low scores win.
• Nested vs. Non-nested: You can only use the Likelihood Ratio Test if one model is a special case of the other. If you are comparing a Lognormal to a Weibull, use AIC or SBC instead!
• Data Size in SBC: Don't forget the \(\ln(n)\) term in SBC. If you forget to factor in the sample size, your calculation will be wrong.
Quick Review Box
1. K-S Test: Looks at the max distance \(|F_n(x) - F^*(x)|\).
2. A-D Test: Like K-S but watches the tails more closely.
3. LRT: Used for nested models; uses \(\chi^2\) distribution.
4. AIC/SBC: Penalize models for being too complex. Smaller = Better.
Keep practicing! Model selection is all about comparing options. Once you get the hang of these three methods, you'll be able to justify exactly why one model is superior to another. You've got this!