Welcome to the Reality Check: Backtesting VaR
Hello there! You’ve already learned how to calculate Value at Risk (VaR). It’s a powerful tool, but how do we know if our VaR model is actually doing its job? This is where Backtesting comes in. Think of backtesting as the "report card" for your risk model. If your model says you should only lose more than \$1 million once a year, but it happens every week, your model is clearly failing. In this chapter, we will learn how to statistically prove whether a model is reliable or needs a total makeover.
\n\nDon’t worry if this seems a bit heavy on the statistics at first! We will break down the math into simple steps and use analogies you can easily remember.
\n\n1. What Exactly is Backtesting?
\nBacktesting is the process of comparing the actual daily profits and losses (P&L) of a portfolio against the VaR estimates generated by a model.
\nWhen an actual loss is greater than the VaR estimate, we call it an exception (or an exceedance). For example, if your 95% 1-day VaR is \$100,000, and you lose \$120,000 today, you have one exception.
Key Idea: The Frequency of Exceptions
If we use a 95% confidence level, we expect to see exceptions 5% of the time. If we look at 100 days of data, we expect around 5 exceptions.
- Too many exceptions? The model is underestimating risk (it's too optimistic).
- Too few exceptions? The model is overestimating risk (it's too conservative).
Quick Review: Backtesting tells us if our "predicted" bad days match our "actual" bad days.
2. The Kupiec POF (Proportion of Failures) Test
The most common way to test frequency is the Kupiec Test. It uses a Binomial Distribution because every day is either a "success" (no exception) or a "failure" (an exception occurred).
The Math (Simplified)
We use a Likelihood Ratio (LR) test to see if the observed number of failures \( x \) is significantly different from the expected number of failures \( N \times p \).
The formula for the Kupiec LR test is:
\( LR_{uc} = -2 \ln \left[ (1-p)^{N-x} p^x \right] + 2 \ln \left[ (1-\frac{x}{N})^{N-x} (\frac{x}{N})^x \right] \)
Wait! Don't let that formula scare you. In the exam, you mainly need to understand that this test checks the Unconditional Coverage. It only cares about the total count of failures, not when they happened.
Did you know?
The Kupiec test is known as an "Unconditional Coverage" test because it doesn't care if all 10 exceptions happened in a single row or were spread out over a year. It just counts them up!
3. Errors in Backtesting: Type I and Type II
Just like in a court of law, we can make two types of mistakes when judging a VaR model. Imagine the model is on trial:
Type I Error: Rejecting a good model.
Analogy: A smoke alarm goes off even though there is no fire. You throw away a perfectly good alarm because of one "false positive."
Type II Error: Failing to reject a bad model.
Analogy: A smoke alarm stays silent while the house is burning. This is much more dangerous in risk management!
Key Takeaway: There is a trade-off. If you make it very hard to commit a Type I error (by being very lenient), you increase the chance of a Type II error (letting a bad model slip through).
4. The Basel "Traffic Light" System
To keep things standardized, the Basel Committee created a simple way to categorize backtesting results based on the number of exceptions observed over 250 business days (one year) for a 99% VaR model.
- Green Zone (0 to 4 exceptions): The model is deemed accurate. No penalty.
- Yellow Zone (5 to 9 exceptions): The model is "maybe" okay, but suspicious. Regulators might increase your capital requirement.
- Red Zone (10+ exceptions): The model is broken. You face an automatic penalty and must fix the model immediately.
Mnemonic Aid: Think of a stoplight.
Green = Go ahead.
Yellow = Slow down and watch out.
Red = Stop! The model is failing.
5. Independence and Clustering: Christoffersen’s Test
Remember how we said the Kupiec test only cares about the total count? That’s a weakness. In the real world, exceptions often come in "clusters" because of market volatility.
Why Clustering is Bad
If you have 5 exceptions spread out over a year, your model might be okay. But if all 5 exceptions happen in one week, your model is failing to adapt to changing market conditions. This is like a weather app that fails to predict every single day of a hurricane—it’s not just wrong; it’s wrong when it matters most.
Christoffersen’s Independence Test
This test specifically checks if today’s exception depends on whether there was an exception yesterday.
- If exceptions are independent, they should appear randomly.
- If exceptions are clustered, the model is missing something important (like volatility autocorrelation).
Conditional Coverage: When you combine the Kupiec test (frequency) and the Christoffersen test (independence), you get a Conditional Coverage test. This is the gold standard for backtesting.
6. Common Pitfalls and Mistakes to Avoid
When studying backtesting, students often trip up on these points:
1. Confusion between Confidence Levels: If the VaR is at 99%, you expect fewer exceptions than at 95%. Always check the confidence level before counting "failures."
2. "Clean" vs. "Dirty" P&L:
- Clean P&L: Uses the same portfolio as yesterday (no new trades). This is better for testing the model's accuracy.
- Dirty P&L: Includes the actual profits/losses from intraday trading. This reflects the firm's actual risk but can hide model flaws.
The FRM curriculum generally prefers Backtesting against Clean P&L to isolate the model's performance.
Summary and Key Takeaways
- Backtesting compares predicted VaR to actual losses.
- Exceptions occur when actual loss > VaR.
- Kupiec Test checks if the number of exceptions is correct (Unconditional Coverage).
- Christoffersen Test checks if exceptions are randomly distributed (Independence).
- Type I Error is rejecting a good model; Type II Error is accepting a bad one.
- Basel Traffic Lights use 250 days of 99% VaR to categorize models into Green, Yellow, and Red zones.
Keep practicing those binomial probability questions! Once you master the logic behind the "Traffic Light" system and the two error types, you'll be well-prepared for this section of the exam.