Bagging, boosting, and random forests

Q: Why are ensemble methods often preferred over single decision trees?

Single decision trees are prone to high variance and instability; ensemble methods combine multiple trees to create a more stable and accurate overall prediction.

Q: What is bootstrapping in the context of ensemble methods?

Bootstrapping is a resampling technique where multiple datasets are created by repeatedly drawing observations from the original training set with replacement.

Q: How does Bagging determine the final prediction for regression and classification tasks?

For regression, Bagging averages the predictions of all trees; for classification, it uses a majority vote (the most frequent class).

Q: What is the main statistical benefit and the primary trade-off of using Bagging?

Bagging significantly reduces variance while keeping bias low, but it sacrifices model interpretability.

Q: What are Out-of-Bag (OOB) observations and how are they useful?

OOB observations are the roughly 1/3 of data points not included in a specific bootstrap sample; they are used to estimate the model's test error without needing a separate validation set.

Q: What is the core difference between Bagging and Random Forests?

Random Forests decorrelate trees by only considering a random subset of predictors at each split, whereas Bagging considers all predictors.

Q: What are the typical values for the number of predictors considered (\( m \)) in a Random Forest split?

Typically, \( m \approx \sqrt{p} \) for classification and \( m \approx p/3 \) for regression, where \( p \) is the total number of predictors.

Q: How does the tree construction process in Boosting differ from Bagging and Random Forests?

Bagging and Random Forests build trees in parallel independently, while Boosting builds trees sequentially, where each tree learns from the previous one.

Q: In Boosting, what do the subsequent trees actually attempt to predict?

Subsequent trees are fit to the residuals (errors) of the current model rather than the original outcome variable.

Q: Why is the number of trees (\( B \)) more critical in Boosting than in Bagging?

Boosting can overfit the data if \( B \) is too large, whereas Bagging and Random Forests generally do not overfit as more trees are added.

Exam SRM – Statistics for Risk Modeling · SOA (Society of Actuaries) · 6 min read

Welcome to the World of Ensemble Methods!

In our previous studies, we learned about single decision trees. They are easy to understand and visualize, but they have a big weakness: they can be very unstable. A small change in your data can result in a completely different tree. This is called high variance.

To fix this, we use Ensemble Methods. Think of an "ensemble" like a choir or an orchestra. One singer might hit a wrong note, but when the whole group sings together, the overall sound is beautiful and stable. In this chapter, we will learn how to combine many trees to create powerful models: Bagging, Random Forests, and Boosting.

1. Bagging (Bootstrap Aggregating)

Bagging is a clever trick to reduce the variance of a statistical learning method. The name comes from combining two words: Bootstrap and Aggregating.

What is Bootstrapping?

Don't worry if this seems tricky! Bootstrapping is just a fancy way of saying "resampling with replacement." Imagine you have a bag of 100 numbered marbles. You pick one, write down the number, and put it back in the bag. You do this 100 times. Some marbles might be picked twice, and some might not be picked at all. This new set of 100 observations is a Bootstrap Sample.

The Bagging Process

1. Create B different bootstrap samples from your training data.
2. Build a deep, unpruned decision tree for each sample.
3. Combine the results:
- For Regression: Average the predictions from all B trees.
- For Classification: Use "majority vote" (the most frequent class predicted).

Why do we do this?

By averaging many high-variance trees, we cancel out the "noise" and get a much more stable prediction. Bagging reduces variance while keeping bias low.

Quick Review: Bagging is great because it makes our predictions more reliable. However, the downside is that we can no longer see a single, simple tree diagram. We trade interpretability for accuracy.

Did you know? On average, a bootstrap sample contains about 2/3 of the original observations. The remaining 1/3 are called Out-of-Bag (OOB) observations. We can use these OOB observations to test the model's accuracy without needing a separate validation set!

2. Random Forests

Random Forests are a "step up" from Bagging. While Bagging is great, it has one flaw: if there is one very strong predictor in the data, most of the trees will use that predictor for the first split. This means all the trees will look very similar (they are correlated).

Averaging similar things doesn't reduce variance as much as averaging different things. Random Forests solve this by "decorrelating" the trees.

How Random Forests Work

Just like Bagging, we build many trees on bootstrapped samples. However, there is a catch: Every time we consider a split in a tree, we are only allowed to choose from a random subset of \( m \) predictors out of the total \( p \) predictors.

Typically, we choose:
- For Regression: \( m \approx p/3 \)
- For Classification: \( m \approx \sqrt{p} \)

Analogy: Imagine you are hiring a team of detectives. In Bagging, every detective looks at the same main clue first. In a Random Forest, you force some detectives to ignore the main clue and look at smaller clues instead. This way, the team discovers things that a single person (or a group of identical thinkers) would miss!

Key Takeaway: If \( m = p \), a Random Forest is exactly the same as Bagging. By choosing \( m < p \), we make the trees different from each other, which reduces variance even further.

3. Boosting

Boosting takes a completely different approach. In Bagging and Random Forests, we build all the trees at the same time (in parallel). In Boosting, we build trees sequentially—one after the other.

The Concept: Learning from Mistakes

In Boosting, each new tree is grown using information from the previous trees. We don't build deep trees; instead, we build very small trees (often called stumps) that slowly improve the model where it's performing poorly.

The Boosting Process (Simplified)

1. Start with a simple model (like the average of the data).
2. Calculate the "errors" (residuals) of the current model.
3. Fit a new small tree to these residuals (not the original outcome).
4. Add this new tree to the model, but only by a small amount (scaled by a learning rate).
5. Repeat this process thousands of times.

Analogy: Think of Boosting like a sculptor. The first tree is a rough cut of the stone. Each following tree is a tiny scrape of the chisel, slowly refining the shape and fixing the imperfections left by the previous steps.

Important Tuning Parameters for Boosting

- Number of Trees (B): Unlike Bagging, Boosting can overfit if \( B \) is too large. We use cross-validation to pick the right number.
- Shrinkage (\( \lambda \)): A small number (like 0.01) that controls the speed of learning. Slow learning generally results in better models.
- Interaction Depth (d): The number of splits in each tree. Often \( d=1 \) (a stump) works remarkably well.

Key Takeaway: Boosting is a "slow learner" that focuses on fixing the errors of its predecessors. It is often the most accurate method but requires careful tuning to avoid overfitting.

4. Variable Importance Measures

Since we are now looking at hundreds of trees, we can't just draw one tree to see what's important. Instead, we use Variable Importance Measures.

- For Regression: We record the total amount that the Residual Sum of Squares (RSS) is decreased due to splits over a given predictor, averaged over all trees. A large value indicates an important predictor.
- For Classification: We do the same, but we measure the decrease in the Gini Index.

Summary Table: Which is Which?

Bagging:
- Trees are built independently (parallel).
- Uses full set of predictors for each split.
- Goal: Reduce variance.

Random Forests:
- Trees are built independently (parallel).
- Uses a random subset of predictors for each split.
- Goal: Decorrelate trees and reduce variance.

Boosting:
- Trees are built sequentially.
- Each tree fits the residuals of the previous model.
- Goal: Slowly reduce bias and variance.

Common Mistake to Avoid: On the exam, don't forget that Bagging and Random Forests do not overfit if you add more trees (B). However, Boosting CAN overfit if the number of trees is too high. Always keep an eye on that shrinkage parameter!

You've got this! These ensemble methods are some of the most powerful tools in a data scientist's kit. Master the logic of "averaging" (Bagging/Forests) vs. "refining" (Boosting), and you'll be well on your way to success in Exam SRM.

Quick check

Can you answer these now?

Open each question to check the key ideas from this chapter.

Why are ensemble methods often preferred over single decision trees?

Single decision trees are prone to high variance and instability; ensemble methods combine multiple trees to create a more stable and accurate overall prediction.

What is bootstrapping in the context of ensemble methods?

Bootstrapping is a resampling technique where multiple datasets are created by repeatedly drawing observations from the original training set with replacement.

How does Bagging determine the final prediction for regression and classification tasks?

For regression, Bagging averages the predictions of all trees; for classification, it uses a majority vote (the most frequent class).

What is the main statistical benefit and the primary trade-off of using Bagging?

Bagging significantly reduces variance while keeping bias low, but it sacrifices model interpretability.

What are Out-of-Bag (OOB) observations and how are they useful?

OOB observations are the roughly 1/3 of data points not included in a specific bootstrap sample; they are used to estimate the model's test error without needing a separate validation set.

What is the core difference between Bagging and Random Forests?

Random Forests decorrelate trees by only considering a random subset of predictors at each split, whereas Bagging considers all predictors.

What are the typical values for the number of predictors considered (\( m \)) in a Random Forest split?

Typically, \( m \approx \sqrt{p} \) for classification and \( m \approx p/3 \) for regression, where \( p \) is the total number of predictors.

How does the tree construction process in Boosting differ from Bagging and Random Forests?

Bagging and Random Forests build trees in parallel independently, while Boosting builds trees sequentially, where each tree learns from the previous one.

In Boosting, what do the subsequent trees actually attempt to predict?

Subsequent trees are fit to the residuals (errors) of the current model rather than the original outcome variable.

Why is the number of trees (\( B \)) more critical in Boosting than in Bagging?

Boosting can overfit the data if \( B \) is too large, whereas Bagging and Random Forests generally do not overfit as more trees are added.

What metrics are used to calculate Variable Importance in ensemble methods?

The total decrease in the Residual Sum of Squares (RSS) for regression or the Gini Index for classification, averaged over all trees.

Ready to test yourself?

Turn these notes into exam-style practice. Get unlimited AI questions on this topic with instant marking and explanations.

Practice This Topic

More Exam SRM – Statistics for Risk Modeling chapters

* The content provided by thinka is generated by AI and may not always be accurate or up-to-date. Please use it as a supplementary resource and verify with official materials.

Put These Notes into Practice

Reading the notes is a great start. Now practise with unlimited AI-generated questions and get instant feedback. 100,000+ students are already improving their grades.

Start Practising Now View Pricing

Done reading? Test yourself with AI practice questions

Practice This Topic Now