Basics of Multiple Regression and Underlying Assumptions

Welcome to Multiple Regression!

In Level I, you learned about Simple Linear Regression—predicting one thing (like a stock's return) using just one other thing (like the market return). But let’s be honest: the world is more complicated than that. A house’s price isn't just about its square footage; it’s also about the neighborhood, the year it was built, and how many bathrooms it has.

Multiple Linear Regression (MLR) is your tool for this "real-world" complexity. It allows us to use two or more independent variables to explain a single dependent variable. Don't worry if the formulas look a bit intimidating at first; we will break them down step-by-step!

1. The Multiple Regression Model

The basic equation for a multiple regression looks like this:

\( Y_i = b_0 + b_1X_{1i} + b_2X_{2i} + ... + b_kX_{ki} + \epsilon_i \)

Where:
- \( Y_i \): The dependent variable (what you are trying to predict).
- \( b_0 \): The intercept (the value of \( Y \) if all \( X \)'s are zero).
- \( b_1, b_2, ..., b_k \): The partial regression coefficients.
- \( X_{1}, X_{2}, ..., X_{k} \): The independent variables (the predictors).
- \( \epsilon_i \): The error term (the "noise" the model can't explain).
- \( k \): The number of independent variables.

Key Concept: Partial Slope Coefficients

This is a favorite topic for exam questions! A coefficient (like \( b_1 \)) tells you how much \( Y \) changes for a one-unit change in \( X_1 \), holding all other independent variables constant. This "holding constant" part is vital. It means we are isolating the unique impact of that one specific variable.

Example: If we predict Salary based on Years of Experience (\( X_1 \)) and Education Level (\( X_2 \)), the coefficient for Experience tells us the salary increase for one more year of work, assuming the person's education level stays exactly the same.

Quick Review:
- Dependent Variable = Effect
- Independent Variables = Causes/Predictors
- Interpreting \( b_j \): The change in \( Y \) per 1-unit change in \( X_j \), ceteris paribus (all else equal).

2. The 6 Underlying Assumptions of MLR

For a regression model to be reliable, it has to follow some rules. If these rules are broken, our conclusions might be "biased" or just plain wrong. Here is the "recipe" for a valid MLR:

1. Linearity: The relationship between the dependent variable and the independent variables is linear. (No curves allowed!)
2. No Multicollinearity: The independent variables are not highly correlated with each other.
3. Homoscedasticity: The variance of the error term is constant across all observations. (If the "spread" of errors changes, we have heteroscedasticity).
4. Independence of Errors: The error term for one observation is not related to the error term of another. (No autocorrelation).
5. Normality: The error terms are normally distributed.
6. Independence of X and Error: The independent variables are not correlated with the error term.

Mnemonic to remember these: "LINE-M"
- Linear relationship
- Independent errors
- Normal errors
- Equal variance (Homoscedasticity)
- Multicollinearity (None!)

Takeaway: If these assumptions hold, our estimators are BLUE (Best Linear Unbiased Estimators).

3. Evaluating the Model: ANOVA and R-Squared

Once we run a regression, how do we know if it’s actually any good? We use the ANOVA (Analysis of Variance) table.

The Three Sums of Squares

1. Total Sum of Squares (SST): The total variation in the dependent variable.
2. Regression Sum of Squares (RSS): The variation explained by your model.
3. Sum of Squared Errors (SSE): The "leftover" variation the model missed.

Relationship: \( SST = RSS + SSE \)

R-Squared (\( R^2 \)) vs. Adjusted R-Squared (\( R^2_{adj} \))

\( R^2 \) (the Coefficient of Determination) tells us the percentage of variation in \( Y \) explained by the independent variables.
\( R^2 = \frac{RSS}{SST} \)

Did you know? If you add *any* new variable to your model, \( R^2 \) will either stay the same or go up. It never goes down! This is a problem because we could just add useless variables to make our model look better.

To fix this, we use Adjusted \( R^2 \):
\( R^2_{adj} = 1 - [(\frac{n-1}{n-k-1}) \times (1 - R^2)] \)

Adjusted \( R^2 \) penalizes you for adding variables that don't add real value. It is always less than or equal to \( R^2 \).

4. Hypothesis Testing: Are the Results Significant?

We use two main tests to see if our model actually tells us something useful.

The t-test (Individual Significance)

Is a specific independent variable useful?
- Null Hypothesis (\( H_0 \)): \( b_j = 0 \) (The variable is useless).
- Alternative Hypothesis (\( H_a \)): \( b_j \neq 0 \) (The variable is significant).
- Calculation: \( t = \frac{\text{Estimated Coefficient}}{\text{Standard Error of Coefficient}} \)

The F-test (Overall Significance)

Do all the independent variables taken together explain the variation in \( Y \)?
- Null Hypothesis (\( H_0 \)): \( b_1 = b_2 = ... = b_k = 0 \) (The whole model is useless).
- Calculation: \( F = \frac{MSR}{MSE} = \frac{RSS/k}{SSE/(n-k-1)} \)

Note: The F-test is always a one-tailed test. If the calculated F is greater than the critical F, we reject the null and say the model is significant.

Common Mistake: Don't confuse the t-test and the F-test. Use the t-test for a single variable and the F-test for the entire group of variables.

5. Standard Error of Estimate (SEE)

The SEE measures the standard deviation of the error terms. It tells us how much the actual data points vary around the regression line.
- A smaller SEE means the model is more "precise" in its predictions.
- Formula: \( SEE = \sqrt{MSE} = \sqrt{\frac{SSE}{n-k-1}} \)

Key Takeaway: Think of SEE as the "average error" of the model. Lower is better!

Summary Checklist for Success

Before moving on to the next chapter, make sure you can:
- [ ] Identify the dependent vs. independent variables.
- [ ] Interpret a coefficient (remember: "all else equal").
- [ ] List the 6 assumptions (LINE-M).
- [ ] Explain why Adjusted \( R^2 \) is often better than \( R^2 \).
- [ ] Calculate a t-statistic and an F-statistic.
- [ ] Understand that \( SST = RSS + SSE \).

Don't worry if these calculations feel heavy! Most CFA questions will provide the ANOVA table for you; your job is to know how the numbers relate to each other and what they mean for your investment analysis. You've got this!