Welcome to Model Evaluation and Cross-Validation!
Hello there! Welcome to one of the most practical chapters in the CS2 Machine Learning syllabus. Up until now, you’ve likely learned how to build models. But how do we know if a model is actually good? If we just test it on the same data we used to build it, the model might look like a superstar, but then fail miserably when we try to use it on real-world insurance claims.
In this chapter, we are going to learn the art of "stress-testing" our models using Cross-validation. We will also learn how to tune the "secret settings" of our models, known as hyper-parameters. Let’s dive in!
1. Parameters vs. Hyper-parameters
Before we evaluate a model, we need to understand what we are actually adjusting. In Machine Learning, there are two types of "knobs" we can turn:
Parameters
These are the internal values the model learns automatically from the data. For example, in a linear regression \( y = \beta_0 + \beta_1 x \), the coefficients \( \beta_0 \) and \( \beta_1 \) are parameters. You don't pick them; the data does.
Hyper-parameters
These are the settings that you (the actuary) must choose before the learning process begins. They control how the model learns.
Analogy: Imagine you are training for a marathon. Your heart rate and muscle strength during the run are parameters (they adapt to the workout). The length of your training plan or the brand of shoes you choose to wear are hyper-parameters (you decide these beforehand to get the best result).
Quick Review:
- Parameters: Learned from data (e.g., weights in a neural network).
- Hyper-parameters: Set by the actuary (e.g., the value of \( k \) in K-nearest neighbors, or the depth of a decision tree).
2. The Golden Rule: Training, Validation, and Testing
To see if our model works on unseen data, we shouldn't use all our data for training. Instead, we split our dataset into three distinct buckets:
- Training Set: The data the model "studies" to learn patterns.
- Validation Set: Used to "tune" the model. We try different hyper-parameters and see which ones perform best on this set.
- Test Set: The "Final Exam." You only use this once, right at the very end, to see how the final model performs on totally fresh data.
Common Mistake to Avoid: Never, ever use your Test Set to make decisions about your model's hyper-parameters! If you do, you are "leaking" information, and your results will be over-optimistic.
Key Takeaway:
Training is for learning, Validation is for choosing the best settings, and Testing is for the final performance check.
3. What is Cross-validation?
If we have a small dataset, splitting it into three parts might leave us with very little data for the model to learn from. Cross-validation is a clever way to use our data more efficiently.
K-fold Cross-validation
This is the most common technique. Here is the step-by-step process:
1. Split the data into \( k \) equal-sized parts (called "folds").
2. Pick one fold to be the Validation Set and use the remaining \( k-1 \) folds as the Training Set.
3. Train the model and calculate the error (e.g., Mean Squared Error) on the validation fold.
4. Repeat this \( k \) times, using a different fold as the validation set each time.
5. Calculate the average of the \( k \) error estimates. This average is your Cross-validation Error.
Don't worry if this seems tricky! Just think of it like a round-robin sports tournament where every team gets a turn to sit out and watch (be the validation set) while the others play (be the training set).
Leave-One-Out Cross-Validation (LOOCV)
This is an extreme version of k-fold cross-validation where \( k \) equals the number of observations (\( n \)). In each turn, you train the model on every single piece of data except one, and then test it on that single omitted observation. This is very accurate but takes a long time to compute if you have a lot of data!
Did you know? Using \( k=5 \) or \( k=10 \) is standard practice in actuarial work. It provides a good balance between computational speed and an accurate estimate of model performance.
4. Using Cross-validation for Hyper-parameter Tuning
How do we find the "best" hyper-parameter? We use a process called Grid Search combined with Cross-validation:
1. Define a list of possible hyper-parameter values (e.g., try \( k=1, 3, 5, 7, 9 \) for a model).
2. For each value, run a K-fold cross-validation.
3. Compare the average error for each value.
4. Choose the value that gave the lowest average error.
Key Takeaway:
Cross-validation helps us choose the hyper-parameters that make our model generalize well to new data, rather than just memorizing the training data.
5. Overfitting and Underfitting
The whole point of cross-validation is to find the "sweet spot" between two extremes:
- Underfitting: The model is too simple (like using a straight line for a curvy pattern). It performs poorly on both training and validation data.
- Overfitting: The model is too complex (it memorizes the noise in the data). It performs perfectly on training data but terribly on validation data.
Memory Aid:
Overfitting is like memorizing the answers to a specific practice exam.
Generalizing (what we want) is like understanding the concepts so you can answer any question.
Summary of Key Points
1. Hyper-parameters are settings chosen by the actuary (not the model).
2. Cross-validation (especially K-fold) is a technique to estimate how a model will perform on unseen data.
3. The Validation set is used to tune hyper-parameters, while the Test set is only for the final evaluation.
4. K-fold involves splitting data into \( k \) parts and rotating which part acts as the "test" data.
5. LOOCV is K-fold where \( k = n \).
6. The Goal: Minimize the cross-validation error to avoid overfitting.
Congratulations! You've just mastered the essentials of evaluating machine learning models for CS2. Keep practicing these concepts, and you'll be well on your way to modeling excellence!