Regression Diagnostics

Welcome to Regression Diagnostics: The "Health Checkup" for Your Models

In your FRM journey so far, you've learned how to build a regression model. But how do we know if that model is actually reliable? In the world of finance, a "broken" model can lead to disastrous risk management decisions. Regression Diagnostics is the process of checking whether our model's assumptions hold true. Think of this chapter as a diagnostic toolkit—much like a mechanic checking an engine—to ensure our results are accurate, unbiased, and ready for use in the real world.

1. Understanding the Goal: Why Diagnostics Matter

For an Ordinary Least Squares (OLS) regression to be the "Best Linear Unbiased Estimator" (BLUE), certain assumptions must be met. If these assumptions are violated, our results might look good on paper but be completely wrong in practice. We are primarily looking for four "troublemakers": Heteroskedasticity, Multicollinearity, Serial Correlation, and Model Specification Errors.

Quick Review: Remember, if a model is "unbiased," it means it hits the target on average. If it is "efficient," it has the smallest possible variance (it’s precise).

2. Heteroskedasticity: The Changing Variance

What is it?
In a perfect world, the "errors" (residuals) of our model should have a constant variance. This is called Homoskedasticity. Heteroskedasticity occurs when the variance of the error terms is not constant.

Analogy: Imagine practicing archery. If you are a consistent shooter, your misses are scattered evenly around the bullseye regardless of how far away you stand. That's homoskedasticity. If you get much sloppier and more erratic the further you stand, your "error variance" is increasing with distance. That’s heteroskedasticity!

Types of Heteroskedasticity

1. Unconditional: The variance changes, but it's not related to our independent variables. This is usually not a major problem for our tests.
2. Conditional: The variance is related to the level of the independent variables. This is the one we worry about in FRM because it invalidates our statistical tests.

Consequences and Detection

Consequences: The coefficients (\( \hat{\beta} \)) are still unbiased, but the Standard Errors are usually underestimated. This makes our t-statistics look larger than they should be, leading us to "find" significant relationships that aren't actually there!

How to spot it:
- Residual Plots: Look for a "fan" or "funnel" shape in the scatter plot of residuals.
- Breusch-Pagan Test: A formal statistical test where we regress the squared residuals on the independent variables.
- White Test: A more robust test that doesn't assume a specific form of heteroskedasticity.

How to fix it: Use White’s Heteroskedasticity-Consistent Standard Errors (also known as robust standard errors). This fixes the t-statistics without changing the coefficients.

Key Takeaway: Heteroskedasticity makes your model look more "certain" than it actually is. Use robust standard errors to stay safe.

3. Multicollinearity: When Variables Are Too Similar

What is it?
Multicollinearity happens when two or more independent variables are highly correlated with each other.

Analogy: Imagine a group project where two students have the exact same knowledge and say the exact same things. It becomes impossible to tell which student actually contributed which idea to the final report. The "contribution" of each is blurred.

Consequences and Detection

Consequences: The model as a whole might look great (high R-squared), but the individual t-stats will be very low. The coefficients become unstable—a small change in data could cause them to flip from positive to negative!

How to spot it:
- High R-squared + Low t-statistics: This is the "classic" red flag.
- Correlation Matrix: Check if the correlation between independent variables is very high (e.g., > 0.8 or 0.9).
- Variance Inflation Factor (VIF): A VIF above 5 or 10 suggests serious multicollinearity.

How to fix it: The simplest fix is to remove one of the highly correlated variables. They are redundant anyway!

Key Takeaway: Multicollinearity doesn't bias your results, but it makes it impossible to tell which variable is actually doing the work.

4. Serial Correlation (Autocorrelation): The Habitual Errors

What is it?
Serial correlation occurs when the error term in one period is related to the error term in a previous period. This is extremely common in Time-Series data.

Analogy: If you are in a bad mood today because you were in a bad mood yesterday, your "mood errors" are serially correlated. They aren't independent events.

Consequences and Detection

Consequences: Similar to heteroskedasticity, OLS coefficients are still unbiased, but standard errors are underestimated. This leads to inflated t-statistics and a false sense of security about your model’s accuracy.

How to spot it:
- Durbin-Watson (DW) Test: This is the most common test.
- If \( DW \approx 2 \): No serial correlation.
- If \( DW < 2 \): Positive serial correlation (most common in finance).
- If \( DW > 2 \): Negative serial correlation.

How to fix it: Use Newey-West adjusted standard errors, which correct for both heteroskedasticity and serial correlation.

Quick Review Table:
Problem: Heteroskedasticity | Fix: White Standard Errors
Problem: Serial Correlation | Fix: Newey-West Standard Errors

5. Model Specification: Building the Right Machine

Sometimes the problem isn't the data, but how we built the model. This is called Specification Error.

Common Errors

1. Omitted Variable Bias: You left out a variable that is actually important and correlated with your independent variables. This does cause bias in your coefficients!
2. Including Irrelevant Variables: You included "junk" variables. This won't cause bias, but it reduces the efficiency (precision) of your model.
3. Functional Form Mis-specification: You used a straight line (linear) to model a relationship that is actually curved (non-linear).

Testing for Normality

We often assume our residuals follow a Normal Distribution. To test this, we use the Jarque-Bera (JB) Test.
The JB test looks at Skewness (symmetry) and Kurtosis (tail thickness). For a normal distribution, Skewness = 0 and Kurtosis = 3. If the JB test statistic is high, we reject the assumption of normality.

Don't worry if this seems tricky at first: Just remember that most financial data has "fat tails" (high kurtosis), so the JB test often fails in the real world!

6. Summary of Key Diagnostic Tests

To help you memorize, here is a quick cheat sheet for your exam prep:

- Heteroskedasticity: Breusch-Pagan or White Test. Fix with White SE.
- Serial Correlation: Durbin-Watson Test. Fix with Newey-West SE.
- Multicollinearity: Look for high R-squared with low t-stats. Fix by removing a variable.
- Normality: Jarque-Bera Test.
- Functional Form: Ramsey RESET test (checks if non-linear components should be added).

Final Encouragement: You don't need to be a math genius to master this! Just focus on identifying the symptoms (the tests) and the cures (the fixes). If you can do that, you'll be well-prepared for the Quantitative Analysis section of the FRM Part I exam.