Welcome to the World of Classifier Performance!

In our journey through Machine Learning within the CS2 curriculum, we’ve learned how to build models that predict outcomes. But how do we know if our model is actually any good? If an actuary builds a model to predict whether a policyholder will lapse (cancel their policy), and the model just says "No" to everyone, it might be 90% accurate because most people don't lapse—but it's completely useless for the business!

In this chapter, we will learn how to look beyond simple accuracy using tools like the Confusion Matrix, Precision, Recall, and the ROC Curve. These metrics help us understand the strengths and weaknesses of our binary classifiers. Don't worry if these terms sound intimidating; we'll break them down step-by-step!

1. The Foundation: The Confusion Matrix

Before we calculate any fancy scores, we need to organize our model's results. A Confusion Matrix is a table that summarizes the performance of a classification algorithm. For a binary classifier (where there are only two possible outcomes: Positive or Negative), the matrix looks like this:

Actual Positive: The event actually happened.
Actual Negative: The event did not happen.
Predicted Positive: Our model said the event would happen.
Predicted Negative: Our model said the event would not happen.

This gives us four quadrants:

  • True Positive (TP): We predicted Positive, and it was Positive. (Success!)
  • True Negative (TN): We predicted Negative, and it was Negative. (Success!)
  • False Positive (FP): We predicted Positive, but it was actually Negative. (This is a Type I Error—think of it as a "False Alarm".)
  • False Negative (FN): We predicted Negative, but it was actually Positive. (This is a Type II Error—think of it as a "Missed Catch".)
The Smoke Alarm Analogy

Think of a smoke alarm as a binary classifier:

  • TP: There is a fire, and the alarm goes off. (Great!)
  • TN: There is no fire, and the alarm stays quiet. (Great!)
  • FP: You burnt some toast (no fire), but the alarm goes off. (Annoying False Alarm.)
  • FN: There is a massive fire, but the alarm stays quiet. (Dangerous Miss!)

Quick Review: The Confusion Matrix isn't a metric itself; it is the data source we use to calculate all the other metrics!

2. Basic Performance Metrics

Now that we have our TP, TN, FP, and FN, we can start calculating scores.

Accuracy

This is the most intuitive metric. It asks: "Overall, how often was the model right?"

\( \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} \)

The Trap: Accuracy can be very misleading if your data is "imbalanced." If 99% of policyholders do not claim, a model that simply predicts "No Claim" for everyone will be 99% accurate but will never help you identify the 1% who actually claim!

Precision

Precision focuses on the Predicted Positives. It asks: "Of all the times the model predicted 'Positive', how many were actually 'Positive'?"

\( \text{Precision} = \frac{TP}{TP + FP} \)

Use this when the cost of a False Positive is high (e.g., if you are accusing someone of insurance fraud, you want to be very sure).

Recall (also called Sensitivity)

Recall focuses on the Actual Positives. It asks: "Of all the 'Actual Positives' that exist, how many did we successfully catch?"

\( \text{Recall} = \frac{TP}{TP + FN} \)

Use this when the cost of a False Negative is high (e.g., missing a critical illness diagnosis in a life insurance application).

Specificity

This is the "Negative" version of Recall. It asks: "Of all the 'Actual Negatives' out there, how many did we correctly identify as Negative?"

\( \text{Specificity} = \frac{TN}{TN + FP} \)

Memory Aid:
Precision is about Predictions (denominator is the predicted row).
Recall is about Reality (denominator is the actual column).

3. The F1 Score: The Balancing Act

Sometimes you want a model that is good at both Precision and Recall. However, there is usually a trade-off: as you try to increase one, the other often drops. The F1 Score is a single number that combines both using the harmonic mean.

\( F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} \)

Why use the harmonic mean? Unlike a simple average, the harmonic mean punishes extreme values. If your Precision is 1.0 but your Recall is 0.0, your F1 score will be 0, not 0.5. This makes it a very robust metric for imbalanced datasets.

Key Takeaway: Use the F1 score when you need a balance between catching as many cases as possible (Recall) while ensuring your predictions are reliable (Precision).

4. Diagnostics: The ROC Curve and AUC

Most classifiers don't just output "Yes" or "No." Instead, they output a probability (e.g., "There is an 85% chance this person will lapse"). To turn that into a "Yes/No" decision, we have to pick a threshold (e.g., anything above 50% is a "Yes").

If we change that threshold, our TP and FP rates change. The ROC Curve (Receiver Operating Characteristic) is a graph that plots:

  • Y-axis: True Positive Rate (Recall)
  • X-axis: False Positive Rate (1 - Specificity)

As you lower the threshold, you catch more Positives (Higher Recall), but you also trigger more False Alarms (Higher False Positive Rate).

Area Under the Curve (AUC)

The AUC is the total area under that ROC curve. It gives us a single number to evaluate the model’s performance across all possible thresholds.

  • AUC = 1.0: A perfect model.
  • AUC = 0.5: No better than a random guess (a diagonal line).
  • AUC < 0.5: The model is actually worse than random!

Did you know? The ROC curve was originally developed during World War II for radar operators to help them distinguish between Japanese aircraft and random noise (like birds)!

5. Summary and Common Mistakes

Common Mistakes to Avoid:
  • Mistaking Recall for Precision: Always check your denominator! Recall is about the actual group; Precision is about the predicted group.
  • Ignoring the context: In the IFoA exams, pay attention to the scenario. If the question asks about a life-threatening disease, Recall is usually more important than Precision. If it's about a low-cost marketing campaign, Precision might matter less.
  • Relying only on Accuracy: Never trust a high accuracy score without checking the balance of the classes.
Quick Review Box:

Confusion Matrix: TP, TN, FP, FN.
Accuracy: Correct / Total.
Precision: TP / (TP + FP).
Recall: TP / (TP + FN).
F1 Score: Balance of Precision and Recall.
ROC Curve: Visualizes the trade-off between Recall and False Alarms.
AUC: The "grade" of the ROC curve (0.5 to 1.0).

You've made it through the core evaluation metrics! These tools are essential for any actuary working with predictive models, ensuring that our risks are measured accurately and our models are reliable.