Evaluating Regression Model Fit and Interpreting Model Results

Welcome to Evaluating Regression Models!

Hello there! If you’ve made it to CFA Level II, you already know that Quantitative Methods can feel like a bit of a mountain to climb. But don’t worry—evaluating a regression model is really just about being a detective. We have a "suspect" (our independent variable) and we want to know: "How well does this suspect explain what happened?"

In this chapter, we aren't just building models; we are checking if they actually work. We will learn how to read the "scorecards" (ANOVA tables), check the "vital signs" of our coefficients, and decide if our model is a masterpiece or just noise. Let’s dive in!

1. The Basics: What are we actually looking at?

When we run a Multiple Linear Regression (MLR), we are looking at an equation that looks like this:
\( Y_i = b_0 + b_1X_{1,i} + b_2X_{2,i} + ... + b_kX_{k,i} + \epsilon_i \)

Key Terms to Remember:
• Dependent Variable (\( Y \)): The thing we are trying to predict (e.g., Stock Returns).
• Independent Variables (\( X \)): The "drivers" or factors we think influence \( Y \) (e.g., Interest Rates, GDP growth).
• Intercept (\( b_0 \)): The value of \( Y \) if all \( X \)s were zero.
• Slope Coefficients (\( b_j \)): How much \( Y \) changes when that specific \( X \) changes by one unit, holding all other variables constant.

Real-World Analogy: Imagine you are predicting the price of a house. The house size (sq ft) and the number of bedrooms are your \( X \) variables. The intercept is the base price of the land, and the slope for "bedrooms" tells you how much the price goes up for every extra bedroom you add, assuming the square footage stays the same.

Summary: Regression helps us quantify the relationship between multiple factors and a single outcome.

2. Assessing Goodness of Fit: Does the Model Work?

Once the computer spits out a regression, we need to know if it's any good. We use a few key metrics for this.

A. The Coefficient of Determination (\( R^2 \))

\( R^2 \) tells us the percentage of the variation in the dependent variable (\( Y \)) that is explained by the independent variables (\( X \)s).
• It ranges from 0 to 1.
• An \( R^2 \) of 0.80 means our model explains 80% of the movement in \( Y \). Awesome!

B. The Problem with \( R^2 \) and the "Adjusted \( R^2 \)" Solution

Did you know? If you keep adding variables to a model—even useless ones like "the number of squirrels in the park"—your \( R^2 \) will almost always go up. This is a trap! To fix this, we use Adjusted \( R^2 \).

Adjusted \( R^2 \) penalizes you for adding variables that don't help much. It is calculated as:
\( \bar{R}^2 = 1 - [(\frac{n-1}{n-k-1}) \times (1-R^2)] \)

Important Rule: Adjusted \( R^2 \) is always less than or equal to \( R^2 \). If you add a variable and Adjusted \( R^2 \) goes down, that variable is "dead weight" and shouldn't be there.

C. Standard Error of Estimate (SEE)

The SEE tells us how far, on average, the actual data points fall from the regression line. Think of it as the "average mistake" the model makes.
• Low SEE = High accuracy (data points are close to the line).
• High SEE = Low accuracy (data points are scattered everywhere).
\( SEE = \sqrt{\frac{SSE}{n-k-1}} \)

Quick Review: \( R^2 \) is "how much we explained," and SEE is "how much we missed."

3. The ANOVA Table: The Model’s Report Card

Don't let the word "ANOVA" (Analysis of Variance) scare you. It’s just a table that breaks down the total variation into "Explained" and "Unexplained" parts.

1. Total Sum of Squares (SST): The total variation in the data.
2. Regression Sum of Squares (RSS): The variation explained by our model.
3. Sum of Squared Errors (SSE): The "leftover" variation we couldn't explain (the noise).

The Golden Rule: \( SST = RSS + SSE \)

Memory Trick: Think of a pizza (SST). The part you ate is RSS (Success!), and the crust you threw away is SSE (Error/Waste).

Key Takeaway: From the ANOVA table, we can calculate \( R^2 \) as \( RSS / SST \).

4. Hypothesis Testing: Are the Results Real?

We use two main tests to see if our results are statistically significant or just lucky guesses.

A. The t-Test (Testing Individual Coefficients)

We use the t-test to see if a single independent variable is actually useful.
Null Hypothesis (\( H_0 \)): The coefficient is zero (\( b_j = 0 \)), meaning it has no effect.
Alternative Hypothesis (\( H_a \)): The coefficient is NOT zero (\( b_j \neq 0 \)).

Test Statistic: \( t = \frac{\hat{b}_j - 0}{s_{\hat{b}_j}} \)
(Coefficient divided by its Standard Error).

Decision Rule: If the absolute value of our calculated \( t \) is greater than the critical t-value (from the table), we reject the null. This means the variable is "statistically significant."

B. The F-Test (Testing the Whole Model)

The F-test checks if at least one of our independent variables is useful. It tests the model as a whole.
Null Hypothesis (\( H_0 \)): All slope coefficients are zero (\( b_1 = b_2 = ... = b_k = 0 \)).
Alternative Hypothesis (\( H_a \)): At least one slope coefficient is not zero.

\( F = \frac{MSR}{MSE} = \frac{RSS/k}{SSE/(n-k-1)} \)

Common Mistake: Students often confuse when to use \( t \) vs \( F \). Remember: T is for Tracking Tiny details (one variable), and F is for the Full Frame (the whole model).

5. Assumptions of Linear Regression

For our regression results to be valid, we have to play by the rules. If these assumptions are broken, our model might lie to us! Use the mnemonic "LINE" to remember them:

• L - Linearity: The relationship between \( X \) and \( Y \) must be a straight line.
• I - Independence: The observations (errors) are not correlated with each other.
• N - Normality: The error terms are normally distributed.
• E - Equal Variance (Homoskedasticity): The "spread" of the errors is constant. They don't get wilder as \( X \) gets bigger.

Summary: If "LINE" holds, our model is usually "BLUE" (Best Linear Unbiased Estimator).

6. Confidence Intervals for Predicted Values

Once we have our model, we can predict a future \( Y \). But because the world is uncertain, we don't just give one number; we give a range.

The Calculation:
\( \hat{Y} \pm (t_c \times s_f) \)
Where \( \hat{Y} \) is our prediction, \( t_c \) is the critical t-value, and \( s_f \) is the standard error of the forecast.

Important Note: The further your \( X \) values are from their historical means, the wider (less certain) your confidence interval becomes. It's harder to predict the unknown extremes than the "normal" middle ground!

Final Quick Review Box

• \( R^2 \): How much of the "story" we explained (0 to 1).
• Adjusted \( R^2 \): Same as \( R^2 \), but penalizes adding useless variables.
• t-test: Checks if 1 variable is significant. (df = \( n - k - 1 \))
• F-test: Checks if the whole model is significant. (df = \( k \) and \( n - k - 1 \))
• SEE: The average "miss" or error of the model.
• Degrees of Freedom (\( df \)): \( n \) is observations, \( k \) is the number of independent variables.

Don't worry if the formulas look intimidating. In the CFA exam, you are more likely to be asked to interpret an ANOVA table or a t-stat than to calculate everything from scratch. Focus on understanding what the numbers are telling you!