Welcome to Regression Trees!

Welcome, future actuaries! Today, we are diving into one of the most intuitive and visual tools in a data scientist's toolkit: Regression Trees. If you’ve ever used a flowchart to make a decision (like "Should I order pizza?"), you already understand the basics of how a tree works. While linear regression tries to fit a straight line through data, regression trees split the data into smaller groups to make predictions. They are easy to explain, fun to visualize, and essential for Exam SRM.

Don't worry if you’ve struggled with heavy math in other chapters—regression trees are much more about logic and "splitting" than complex calculus!

1. What is a Regression Tree?

A regression tree is a type of Decision Tree used when the target variable (the thing we want to predict) is quantitative (numerical), like the price of a house or the amount of an insurance claim.

Key Terminology

To talk like a pro, you need to know the "anatomy" of the tree:

  • Root Node: The very top of the tree where the first split happens. It contains the entire dataset.
  • Internal Nodes: The points where the tree branches out based on a "yes/no" or "true/false" condition.
  • Branches: The segments connecting nodes.
  • Leaf Nodes (Terminal Nodes): The ends of the tree. This is where the final prediction lives!

How Predictions are Made

In a regression tree, every observation that ends up in the same Leaf Node gets the same prediction. That prediction is simply the mean (average) of the response values of the training data in that specific leaf.

Example: If a leaf node contains three houses sold for \$200k, \$210k, and \$220k, any new house falling into that leaf will be predicted at \$210k.

Quick Review:
- Regression Tree: Predicts numbers.
- Prediction: Average of the observations in that leaf.

2. Building the Tree: Recursive Binary Splitting

How does the computer decide where to split the data? It uses a method called Recursive Binary Splitting. Let’s break those fancy words down:

  • Binary: Each split results in exactly two branches.
  • Recursive: The process repeats over and over for each new branch.
  • Greedy: At every step, the algorithm chooses the best split right now, without looking ahead to see if a different split would be better in the long run.

The Math: Minimizing RSS

The goal of every split is to make the groups as "pure" as possible. We do this by minimizing the Residual Sum of Squares (RSS). We want the data points in each group to be as close to the average of that group as possible.

The formula for RSS in a tree with \( J \) terminal nodes (regions \( R_1, R_2, ..., R_J \)) is:

\( RSS = \sum_{j=1}^{J} \sum_{i \in R_j} (y_i - \hat{y}_{R_j})^2 \)

Where \( \hat{y}_{R_j} \) is the mean response for the training observations within the \( j \)-th leaf.

Step-by-Step Splitting:

  1. Start with all data in one group.
  2. Look at every possible predictor \( X_j \) and every possible split point \( s \).
  3. Choose the predictor and split point that results in the lowest RSS.
  4. Repeat the process for the resulting two regions.

Key Takeaway: We split the data into rectangles (or boxes) in the feature space. We are always trying to find the split that reduces the error (RSS) the most at that specific moment.

3. The Problem of Overfitting

If we let a tree grow forever, it will eventually have one leaf for every single data point. This would result in an RSS of zero on our training data, but it would be terrible at predicting new data! This is called overfitting.

Analogy: Imagine memorizing every single question and answer in a practice exam instead of learning the concepts. You’d get 100% on the practice, but you’d fail the real exam because you can’t handle new questions!

To prevent this, we have two options:
1. Stop growing the tree early (often doesn't work well).
2. Pruning the tree (the preferred method).

4. Tree Pruning & Cost-Complexity Pruning

Pruning means "cutting back" the branches of a large tree to find a smaller subtree that performs better on new data. But how do we know which branches to cut?

Cost-Complexity Pruning (Weakest Link Pruning)

We use a tuning parameter called \( \alpha \) (alpha). We want to minimize a score that balances the fit of the tree with its complexity:

\( \sum_{m=1}^{|T|} \sum_{i: x_i \in R_m} (y_i - \hat{y}_{R_m})^2 + \alpha |T| \)

Let's simplify this: Total Score = RSS + (Alpha × Number of Leaves)

  • RSS: Measures how well the tree fits the training data (Error).
  • \( |T| \): The number of leaf nodes (Complexity).
  • \( \alpha \): The "penalty" for having a complex tree.

How \( \alpha \) works:
- If \( \alpha = 0 \): The penalty is zero. We get the big, overfitted tree.
- As \( \alpha \) increases: The penalty for having leaves grows. We "prune" the tree back to a smaller size.
- We usually find the best \( \alpha \) using K-fold Cross-Validation.

Did you know? This is very similar to Lasso Regression! Both use a penalty term to simplify the model and prevent overfitting.

5. Pros and Cons of Regression Trees

It's important for the SRM exam to know why we would (or wouldn't) choose a tree over a linear model.

Advantages (The Good Stuff):

  • Easy to Explain: You can show a tree to a non-actuary, and they will understand it immediately.
  • Handles Non-Linearity: Trees don't assume a straight-line relationship.
  • No Dummy Variables Needed: Trees handle categorical predictors (like "Red", "Blue", "Green") naturally without needing special coding.
  • Mirror Human Decision Making: We naturally think in "if-then" steps.

Disadvantages (The Tough Stuff):

  • Lower Predictive Accuracy: A single tree usually isn't as accurate as a linear regression or more complex methods (like Random Forests).
  • High Variance (Instability): A small change in the data can result in a completely different tree.
  • The "Boxy" Limitation: Since splits are always perpendicular to the axes, trees struggle to model relationships that are truly diagonal or smooth curves.

Memory Trick: Think of a tree as a "Rough Draft." It's great for getting a general idea and explaining things simply, but you might need a more polished "Final Draft" (like Random Forests or Boosting) for high-stakes predictions.

6. Summary and Final Tips

Common Mistake to Avoid: On the exam, don't confuse Regression Trees with Classification Trees.
- Regression: Predicting a number. Uses RSS. Prediction = Mean.
- Classification: Predicting a category. Uses Gini Index or Entropy. Prediction = Mode (Most common category).

Key Takeaways:

1. Trees use Recursive Binary Splitting to minimize RSS.
2. They are Top-Down and Greedy.
3. Pruning via Cost-Complexity (using \( \alpha \)) prevents overfitting.
4. Trees are high variance—they change easily with new data—but are highly interpretable.

Don't worry if this seems tricky at first! Just remember that at its heart, a regression tree is just a series of "yes/no" questions designed to group similar numbers together. Keep practicing those practice problems, and you'll be a tree expert in no time!