Welcome to Backtesting and Simulation!
Hello there! You’ve reached a crucial part of the Portfolio Management curriculum. Think of this chapter as the "stress test" for investment ideas. Before a fund manager bets millions of dollars on a new strategy, they need to know: "Would this have worked in the past?" and "What happens if things go wrong in the future?"
In this module, we will learn how to look backward using Backtesting and how to look forward using Simulation. Don't worry if these terms sound technical—we'll break them down step-by-step with simple analogies. Let’s dive in!
1. What is Backtesting?
Backtesting is the process of applying an investment strategy or analytical model to historical data to see how it would have performed. It’s like "paper trading" in reverse.
The Core Goal: To verify if the logic behind a strategy is sound before risking actual capital.
The Step-by-Step Backtesting Process
1. Strategy Design: Define your rules. Example: "Buy stocks with a P/E ratio below 15 and sell them when it hits 25."
2. Historical Data Collection: Gather price and financial data for the period you want to test (e.g., 2010–2023).
3. Simulation of Trades: Run your rules through that data. The computer "buys" and "sells" based on your rules.
4. Performance Evaluation: Look at the results. Did you make money? How much risk did you take?
Quick Review: Backtesting tells us "What would have happened?" based on "What did happen."
2. Key Performance Metrics
How do we know if a backtest was successful? We use specific metrics to measure "risk-adjusted" performance. It’s not just about the total return; it’s about how much "heartburn" you had to endure to get that return.
A. The Sharpe Ratio
This measures excess return per unit of total risk (standard deviation).
\( \text{Sharpe Ratio} = \frac{R_p - R_f}{\sigma_p} \)
Where \( R_p \) is portfolio return, \( R_f \) is the risk-free rate, and \( \sigma_p \) is the portfolio's standard deviation.
B. Maximum Drawdown (MDD)
This is the "ouch" factor. It measures the largest percentage drop from a peak to a trough before a new peak is achieved.
Analogy: If your account went from \$100 down to \$60, then back up to \$110, your MDD was 40%.
C. Information Ratio (IR)
This is crucial for active managers. It measures the "active return" (return above a benchmark) per unit of "active risk" (tracking error).
\( \text{Information Ratio} = \frac{R_p - R_b}{\sigma_{(R_p - R_b)}} \)
Key Takeaway: A good strategy has a high Sharpe Ratio and a manageable Maximum Drawdown. If the drawdown is too large, investors will panic and quit before the strategy can recover!
3. Common Pitfalls: Why Backtests Fail
This is a favorite topic for CFA examiners! A backtest might look amazing on paper but fail miserably in real life. Why? Usually because of one of these "biases."
1. Survivorship Bias
This happens when you only test your strategy on companies that exist today. You forget about the companies that went bankrupt or were delisted during your test period.
Example: Testing a "tech strategy" from 1999–2005 but only using companies currently in the S&P 500. You are ignoring all the "dot-coms" that failed!
2. Look-Ahead Bias
Using information in your test that wouldn't have been available at the time.
Example: Testing a strategy that "buys" a stock based on its year-end earnings on January 1st. In reality, year-end earnings aren't reported until February or March!
3. Data Mining (Overfitting)
If you torture the data long enough, it will confess to anything. This is searching through thousands of variables until you find a random pattern that worked in the past but has no economic logic.
Analogy: If you find that "stocks go up when it rains in Tuesday," that's probably a coincidence, not a strategy.
4. Transaction Costs & Liquidity
Many backtests assume you can buy and sell for free. In reality, trading commissions and bid-ask spreads eat into your profits.
Memory Trick: Think of S.L.O.T. (Survivorship, Look-ahead, Overfitting, Transaction costs). If your backtest has these, you're "slotting" your money into a losing machine!
4. Simulation: Monte Carlo and Historical
While backtesting looks at one specific path the past took, Simulation explores many possible paths the future might take.
A. Monte Carlo Simulation (MCS)
Monte Carlo involves using a computer to generate thousands of random "what if" scenarios based on statistical distributions (like the Normal Distribution).
How it works: You tell the computer the average return and volatility of an asset, and it "rolls the dice" 10,000 times to show you the range of possible outcomes.
Advantages of Monte Carlo:
- It shows a distribution of outcomes (not just one number).
- It’s great for "Path Dependency" (e.g., will I run out of money during retirement if the market crashes in Year 2?).
- It can model complex "optionality."
Disadvantages:
- "Garbage in, garbage out." If your input assumptions (mean and variance) are wrong, the results are useless.
- It often assumes a Normal Distribution, which might ignore "Fat Tails" (extreme market crashes).
B. Historical Simulation
Instead of using random numbers, you use actual past changes in market rates and prices. You "re-play" the past but apply it to your current portfolio.
Difference: Unlike Monte Carlo, Historical Simulation doesn't require a "distribution" (like a Bell Curve). It uses the actual distribution of the past, which naturally includes "fat tails" or market crashes.
Did you know? Monte Carlo is named after the famous casino in Monaco. It’s all about probability and randomness!
5. Sensitivity Analysis vs. Scenario Analysis
These are "Simulation-lite" tools that help us understand risk.
Sensitivity Analysis: Changing one variable at a time (e.g., "What happens if interest rates rise by 1%?") to see how it affects the portfolio.
Scenario Analysis: Changing multiple variables at once to reflect a specific event (e.g., "What happens if there is a global pandemic?"). This involves changes in GDP, interest rates, and consumer spending simultaneously.
Key Takeaway: Sensitivity is for small "tweaks." Scenario is for "big events."
6. Summary and Final Tips
Don't worry if this seems like a lot to memorize. Just remember these three "Big Ideas":
1. Backtesting is a historical check, but it is prone to Biases (Survivorship and Look-ahead are the big ones).
2. Monte Carlo uses random distributions to model thousands of future possibilities.
3. The Sharpe Ratio and Max Drawdown are your primary tools for judging if a strategy's return is worth the risk.
Common Mistake to Avoid: On the exam, if a question says a model works perfectly because it was tested on 50 years of data, be suspicious! It might be Overfitted or suffering from Data Mining. More data isn't always better if you're just looking for coincidental patterns.
Keep pushing forward! You've got this. Understanding how to test and simulate strategies is what separates a professional portfolio manager from a lucky gambler.