Bagging and random forests

Q: Why are single decision trees considered to have high variance?

They are "fragile" because small changes in the training data can result in a significantly different tree structure and predictions.

Q: What is the definition of Bagging?

Bagging, short for Bootstrap Aggregating, is an ensemble technique that trains multiple models on different bootstrap samples and combines their results.

Q: What does it mean to sample with replacement (Bootstrapping)?

It is a sampling method where each observation is returned to the population after being selected, allowing it to potentially appear multiple times in the same sample.

Q: How are final predictions calculated in Bagging for regression and classification?

Regression predictions are averaged across all trees, while classification predictions are determined by a majority vote.

Q: What is the primary statistical benefit of Bagging?

Bagging reduces the model's variance without significantly increasing its bias.

Q: What are Out-of-Bag (OOB) observations?

They are the observations in the original dataset that were not selected for a particular bootstrap sample, typically accounting for about 1/3 of the data.

Q: How is the OOB Error calculated?

For each observation, a prediction is made using only the trees where that observation was "out-of-bag," and the error is then averaged across the entire dataset.

Q: What is the problem of "Tree Correlation" in standard Bagging?

If a very strong predictor exists, most bagged trees will look similar because they all split on that variable first, which limits the benefit of averaging.

Q: How do Random Forests solve the problem of tree correlation?

They introduce feature randomness by only allowing the model to consider a random subset of predictors at each split.

Q: What are the standard rules of thumb for the mtry parameter (m)?

For classification, \( m \approx \sqrt{p} \); for regression, \( m \approx p/3 \).

Exam PA – Predictive Analytics · SOA (Society of Actuaries) · 6 min read

Welcome to the World of Ensemble Learning!

In our previous studies, we learned about decision trees. While they are easy to understand, they have a big weakness: they are "fragile." If you change the data just a little bit, you might get a completely different tree. In technical terms, we say they have high variance.

In this chapter, we are going to learn how to fix that by using "Ensemble Methods." Instead of relying on one single tree, we will build a whole forest! We’ll focus on two main techniques: Bagging and Random Forests. These methods are some of the most powerful tools in an actuary's predictive modeling toolkit.

1. The Foundation: Bagging (Bootstrap Aggregation)

Don't worry if the name sounds strange! Bagging is just a shorthand for Bootstrap Aggregating. The logic is simple: "The wisdom of the crowd is better than the opinion of one expert."

What is Bootstrapping?

Imagine you have a bag of 100 marbles. You want to create a "new" bag of 100 marbles. You pick one out, write down its color, and put it back in. You repeat this 100 times. Because you put the marbles back, some marbles might appear twice or three times, and some might not appear at all. This is called sampling with replacement.

How Bagging Works Step-by-Step:

1. Bootstrap: We take our original dataset and create many different bootstrap samples (usually hundreds).
2. Train: We grow a full, deep decision tree on each of these samples. Because the samples are slightly different, the trees will be slightly different too.
3. Aggregate: To make a final prediction, we combine the results from all the trees:
• For Regression (predicting a number): Take the average of all the trees' predictions.
• For Classification (predicting a category): Take a "majority vote." Whatever category most trees pick is our final answer.

Why do we do this?

The main goal of Bagging is to reduce variance. By averaging many trees, we cancel out the "noise" and errors of individual trees, leading to a much more stable and accurate prediction. It’s like asking 100 people to guess the weight of an elephant; individual guesses might be way off, but the average is usually very close!

Quick Review: Bagging = Bootstrap (sampling with replacement) + Aggregating (averaging the results). It lowers variance without making the bias worse.

2. Out-of-Bag (OOB) Error Estimation

Did you know? When we use Bagging, we don't actually need to use a separate "Validation Set" to see how well our model is doing. We can use the Out-of-Bag (OOB) observations.

When we create a bootstrap sample, on average, about one-third (1/3) of the data is left out. These left-out rows are called "out-of-bag" observations. Since the tree never saw this data during training, we can use it as a "mini test set."

We predict the value for each row using only the trees that didn't use that row for training. This gives us the OOB Error, which is a very reliable estimate of how the model will perform on brand-new data.

3. Taking it Further: Random Forests

Random Forests are a clever improvement over Bagging. To understand why we need them, we have to look at a sneaky problem in Bagging called Tree Correlation.

The Problem: The "Strong Predictor" Trap

In regular Bagging, if there is one very strong predictor (like "Age" in a life insurance model), almost every tree will use that predictor for the very first split. This means all the trees will look very similar. If the trees are similar, averaging them doesn't help as much—it's like asking 100 people for their opinion, but they all read the same newspaper article before answering.

The Solution: Feature Randomness

Random Forests force the trees to be different (to "de-correlate" them). When building a tree in a Random Forest:
1. At each split, the model is only allowed to choose from a random subset of predictors.
2. If we have \( p \) total predictors, we usually only allow the model to look at \( m \) predictors at each split, where \( m < p \).

Standard rules of thumb for \( m \):
• For Classification: \( m \approx \sqrt{p} \)
• For Regression: \( m \approx p/3 \)

By forcing the model to ignore the strongest predictor occasionally, it is forced to find patterns in the "weaker" predictors that it would have otherwise ignored. This makes the forest much more diverse and powerful.

Analogy: Imagine a talent show. In Bagging, every judge looks at the same performance. In Random Forests, one judge is told "You can't look at the singer's voice, only their dance," and another is told "You can't look at their dance, only their costume." By forcing them to focus on different things, the final combined score is more well-rounded.

4. Variable Importance Measures

One downside of Bagging and Random Forests is that they are "Black Box" models. It's hard to look at 500 trees and explain exactly how the model works. However, we can still find out which variables are the most important.

1. Mean Decrease in Gini Index (for Classification): This measures how much the "purity" of the nodes increases when a specific variable is used for splitting. A high value means the variable is very useful for sorting observations into classes.
2. Mean Decrease in RSS (for Regression): This measures how much the Total Sum of Squares (RSS) drops when a specific variable is used. A bigger drop means the variable is important for predicting the numerical outcome.

Key Takeaway: Even if we can't see the "logic" of the forest as easily as a single tree, Variable Importance plots tell us which features are doing the heavy lifting.

5. Summary and Common Pitfalls

Common Mistake to Avoid: Students often worry that adding too many trees to a Random Forest will cause overfitting. Actually, Random Forests generally do not overfit just because you add more trees. Adding more trees just makes the variance settle down. The main downside to "too many trees" is simply that the computer takes longer to run the model!

Quick Summary Checklist:

• Bagging reduces variance by averaging many trees grown on bootstrap samples.
• Random Forests improve on Bagging by only allowing a random subset of predictors at each split (this is the mtry parameter in R).
• mtry is the most important hyperparameter to tune. If \( m = p \), Random Forest is just regular Bagging.
• OOB Error allows us to validate the model without a separate test set.
• Variable Importance helps us explain which features matter most in these complex models.

Keep going! You're doing great. Tree-based models are a huge part of Exam PA, and mastering Random Forests is a giant step toward passing!

Quick check

Can you answer these now?

Open each question to check the key ideas from this chapter.

Why are single decision trees considered to have high variance?

They are "fragile" because small changes in the training data can result in a significantly different tree structure and predictions.

What is the definition of Bagging?

Bagging, short for Bootstrap Aggregating, is an ensemble technique that trains multiple models on different bootstrap samples and combines their results.

What does it mean to sample with replacement (Bootstrapping)?

It is a sampling method where each observation is returned to the population after being selected, allowing it to potentially appear multiple times in the same sample.

How are final predictions calculated in Bagging for regression and classification?

Regression predictions are averaged across all trees, while classification predictions are determined by a majority vote.

What is the primary statistical benefit of Bagging?

Bagging reduces the model's variance without significantly increasing its bias.

What are Out-of-Bag (OOB) observations?

They are the observations in the original dataset that were not selected for a particular bootstrap sample, typically accounting for about 1/3 of the data.

How is the OOB Error calculated?

For each observation, a prediction is made using only the trees where that observation was "out-of-bag," and the error is then averaged across the entire dataset.

What is the problem of "Tree Correlation" in standard Bagging?

If a very strong predictor exists, most bagged trees will look similar because they all split on that variable first, which limits the benefit of averaging.

How do Random Forests solve the problem of tree correlation?

They introduce feature randomness by only allowing the model to consider a random subset of predictors at each split.

What are the standard rules of thumb for the mtry parameter (m)?

For classification, \( m \approx \sqrt{p} \); for regression, \( m \approx p/3 \).

How is Variable Importance measured in Random Forests?

It is measured by the Mean Decrease in RSS for regression or the Mean Decrease in Gini Index for classification.

Does adding more trees to a Random Forest cause overfitting?

No; increasing the number of trees does not lead to overfitting, it only causes the variance to settle and improves model stability.

What is the relationship between Bagging and Random Forests regarding the mtry parameter?

Bagging is a special case of a Random Forest where the number of predictors considered at each split (m) is equal to the total number of predictors (p).

Ready to test yourself?

Turn these notes into exam-style practice. Get unlimited AI questions on this topic with instant marking and explanations.

Practice This Topic

More Exam PA – Predictive Analytics chapters

* The content provided by thinka is generated by AI and may not always be accurate or up-to-date. Please use it as a supplementary resource and verify with official materials.

Put These Notes into Practice

Reading the notes is a great start. Now practise with unlimited AI-generated questions and get instant feedback. 100,000+ students are already improving their grades.

Start Practising Now View Pricing

Done reading? Test yourself with AI practice questions

Practice This Topic Now