Classification trees

Welcome to the World of Classification Trees!

In our previous studies, we looked at Regression Trees, which help us predict numerical values like the price of a house or the amount of an insurance claim. But what if we want to predict a category? For example, "Will this driver file a claim?" (Yes/No) or "Is this tumor malignant or benign?"

That’s where Classification Trees come in! Think of a classification tree as a series of questions that help you sort data into the correct "bucket." They are incredibly visual, easy to explain to non-actuaries, and form the foundation for more complex models you'll see later in Exam SRM. Let’s dive in!

1. What is a Classification Tree?

A Classification Tree is used to predict a qualitative (categorical) response rather than a quantitative one.

In a classification tree, we predict that each observation belongs to the most commonly occurring class of training observations in the region to which it belongs. This is also known as the plurality rule or the mode.

Real-World Analogy: Imagine you are sorting fruit. Your first question might be "Is it round?" then "Is it red?" By the end of these questions, you’ve classified the fruit as an "Apple." You didn't calculate a number; you found a category.

Quick Review: - Regression Tree: Predicts a mean value (e.g., $5,000). - Classification Tree: Predicts a category (e.g., "High Risk").

2. Building the Tree: Recursive Binary Splitting

Just like regression trees, we build classification trees using Recursive Binary Splitting. We start at the top and split the data into two branches based on a predictor variable, then repeat the process for each branch.

However, we have a problem: In regression, we used RSS (Residual Sum of Squares) to decide where to split. You can't subtract "Apple" from "Orange," so RSS doesn't work here! Instead, we need a way to measure node purity.

Why "Purity" Matters

We want our resulting groups (nodes) to be as "pure" as possible. A pure node is one where almost all observations belong to the same category. If a node is "impure," it means it’s a messy mix of different categories.

3. Measuring Impurity: The Three Key Metrics

To decide where to make a split, we look for the split that results in the lowest impurity. There are three common ways to measure this in the SRM curriculum:

A. Classification Error Rate

This is the simplest measure. It is the fraction of the training observations in a region that do not belong to the most common class.

\( E = 1 - \max_k(\hat{p}_{mk}) \)

Where \( \hat{p}_{mk} \) represents the proportion of training observations in the m-th region that are from the k-th class.

Common Pitfall: While simple, the classification error rate is not sensitive enough for tree-growing. It’s better for pruning the tree later, but not for picking the initial splits.

B. Gini Index

The Gini Index is a measure of total variance across the categories. It is defined as:

\( G = \sum_{k=1}^K \hat{p}_{mk}(1 - \hat{p}_{mk}) \)

Key Concept: If all observations in a node belong to one class (pure), the Gini index will be zero. If the classes are evenly split (impure), the Gini index will be high. We want to minimize the Gini index.

C. Entropy (Deviance)

Entropy is a concept borrowed from information theory. It measures the "disorder" in a node. It is defined as:

\( D = -\sum_{k=1}^K \hat{p}_{mk} \log \hat{p}_{mk} \)

Similar to the Gini index, if a node is pure, the Entropy will be zero. If a node is messy and mixed, Entropy will be high.

Memory Aid: Think of a room full of toddlers. - Pure: Everyone is playing quietly with blocks (Low Entropy/Gini). - Impure: Some are screaming, some are sleeping, some are throwing blocks (High Entropy/Gini).

Summary Table of Impurity Measures

1. Classification Error: Good for final evaluation, bad for splitting.
2. Gini Index: Excellent for splitting; measures "purity."
3. Entropy: Excellent for splitting; measures "disorder."

4. Comparing Gini and Entropy

Don't stress too much about the mathematical difference between Gini and Entropy—they are actually quite similar! In practice, they usually produce very similar trees. Both are more sensitive to node purity than the simple classification error rate.

Did you know? When building a tree, the software calculates the Gini or Entropy for every possible split and chooses the one that reduces the impurity the most. This is the "Greedy" approach!

5. Interpretation and Prediction

Once the tree is built, how do we use it?

1. Take a new observation.
2. Follow the "Yes/No" questions down the tree until you reach a Leaf Node (terminal node).
3. The predicted class is the most common class in that leaf node.

Example: If a leaf node contains 80 "Safe" drivers and 20 "At-Risk" drivers, any new person landing in that leaf node is predicted as "Safe."

Wait! What about probabilities?
A classification tree can also predict the probability of belonging to a class. In the example above, we would say there is an 80% probability the person is a "Safe" driver.

6. Pros and Cons of Classification Trees

It’s important for the SRM exam to know when to use trees and when they might fail.

Advantages:
- Easy to explain: You can show a picture of the tree to a manager, and they will understand it immediately.
- No Dummy Variables needed: Trees handle categorical predictors (like "Color") naturally without needing to turn them into 0s and 1s.
- Mimics human decision-making: We often think in "If-Then" statements.

Disadvantages:
- High Variance: A small change in the data can result in a completely different tree. (This is their biggest weakness!)
- Lower Predictive Accuracy: A single tree often isn't as accurate as other models (like GLMs). Note: This is why we eventually learn about "Forests" and "Boosting" to fix this!

7. Key Takeaways for the Exam

1. Prediction Rule: We predict the mode (most common class) of the training data in a region.
2. Splitting Criteria: We use Gini Index or Entropy to grow the tree because they favor pure nodes.
3. Error Rate: The Classification Error Rate is typically used for pruning/evaluating, not for the initial splitting.
4. Interpretability: Trees are highly interpretable but often suffer from high variance.

Don't worry if the math for Entropy looks scary! On the exam, you are more likely to be asked about the concept of purity or to perform a simple Gini calculation than to derive complex logarithms. Focus on understanding why we want pure nodes!