Regression with Multiple Explanatory Variables

Welcome to Multiple Regression!

In our previous studies, we looked at Simple Linear Regression (SLR), where we used one variable (like the market return) to predict another (like a stock's return). But let’s be honest—the world is rarely that simple! A stock's price isn't just affected by the market; it’s affected by interest rates, oil prices, and company earnings all at once.

In this chapter, we step into the real world by learning Multiple Linear Regression (MLR). This allows us to use several explanatory variables to explain a single dependent variable. Don't worry if this seems a bit math-heavy at first; we will break it down piece by piece. By the end of these notes, you'll see that it’s just an extension of what you already know!

1. The Multiple Regression Model

In a multiple regression, we are trying to find the "best fit" line through a multi-dimensional cloud of data points. The mathematical formula looks like this:

\( Y_i = \beta_0 + \beta_1 X_{1i} + \beta_2 X_{2i} + ... + \beta_k X_{ki} + \epsilon_i \)

Where:
- \( Y_i \): The dependent variable (what we want to predict).
- \( \beta_0 \): The intercept (the value of \(Y\) if all \(X\)s are zero).
- \( \beta_1, \beta_2... \): The partial regression coefficients.
- \( X_1, X_2... \): The independent (explanatory) variables.
- \( \epsilon_i \): The error term (everything else that affects \(Y\) but isn't in our model).

Understanding the "Partial" Effect

The most important thing to remember for the FRM exam is how to interpret these coefficients. We use the phrase "ceteris paribus" (all else being equal).

Example: Suppose we are predicting a house price (\(Y\)) based on square footage (\(X_1\)) and the number of bedrooms (\(X_2\)). If \(\beta_1 = 150\), it means that for every 1-unit increase in square footage, the house price increases by $150, holding the number of bedrooms constant.

Quick Review: In multiple regression, each coefficient tells us the effect of that specific variable while keeping all other variables in the model unchanged.

2. Goodness of Fit: \(R^2\) vs. Adjusted \(R^2\)

In Simple Regression, we used \(R^2\) to see how well our model explained the data. In Multiple Regression, there is a "trap" with the standard \(R^2\).

The Problem: Every time you add a new variable to your model—even if it is totally useless (like using "the number of birds in the sky" to predict "stock prices")—the \(R^2\) will always stay the same or go up. It never goes down!

The Solution: Adjusted \(R^2\)

To fix this, we use the Adjusted \(R^2\) (denoted as \(\bar{R}^2\)). This version "penalizes" you for adding variables that don't actually help explain the dependent variable. It helps us decide if adding a new variable is actually worth it.

\( \bar{R}^2 = 1 - \frac{(1-R^2)(n-1)}{n-k-1} \)

Where:
- \( n \) = number of observations
- \( k \) = number of independent variables

Did you know? Because of the penalty for adding variables, the Adjusted \(R^2\) will always be less than or equal to the standard \(R^2\). If you see an exam question where Adjusted \(R^2 > R^2\), you know it’s a trick!

Key Takeaway: Use \(R^2\) to see how much variance is explained, but use Adjusted \(R^2\) when comparing models with different numbers of variables.

3. Hypothesis Testing: T-tests and F-tests

Now that we have our model, we need to know if our variables actually matter. We use two main types of tests:

The T-test (Individual Significance)

We use a t-test to check if a single variable is statistically significant.
Null Hypothesis (\(H_0\)): \(\beta_i = 0\) (The variable has no effect).
Formula: \( t = \frac{\text{Estimated Coefficient} - \text{Hypothesized Value}}{\text{Standard Error of Coefficient}} \)

The F-test (Joint Significance)

The F-test checks if all the explanatory variables together as a group have any predictive power. It tests if at least one of the \(\beta\) coefficients is not zero.
Null Hypothesis (\(H_0\)): \(\beta_1 = \beta_2 = ... = \beta_k = 0\).
Analogy: If a t-test is like checking if an individual player on a team is good, the F-test is like checking if the whole team is capable of winning a game.

Common Mistake: Students often think that if all individual t-tests fail, the F-test must also fail. This isn't always true! Sometimes variables work better together than they do alone.

4. Pitfalls: Omitted Variable Bias and Multicollinearity

Multiple regression is powerful, but it has two major "enemies" you must recognize for the FRM exam.

Enemy #1: Omitted Variable Bias (OVB)

This happens when you leave out a variable that is important (it affects \(Y\)) and correlated with one of the variables already in your model.

The Result: Your estimated coefficients will be biased and inconsistent. The model "assigns" the credit (or blame) for the missing variable to the variables you did include.

Enemy #2: Multicollinearity

This happens when two or more of your "independent" variables are highly correlated with each other. For example, if you try to predict a person's height using "Length of left leg" and "Length of right leg."

How to spot it:
1. The \(R^2\) is very high, and the F-test is significant.
2. BUT, the individual t-tests are not significant (the variables look "useless" individually).
3. The standard errors of the coefficients are very large.

Analogy: Imagine two people shouting the same message at the same time. You know the message is being sent (high \(R^2\)), but you can't tell which person is responsible for which part of the message (low t-stats).

Quick Review: OVB makes your results wrong (biased). Multicollinearity makes your results unreliable (high variance/standard errors).

5. Assumptions of the Multiple Regression Model

For our OLS (Ordinary Least Squares) estimates to be the "Best Linear Unbiased Estimator" (BLUE), certain assumptions must hold. These are the Gauss-Markov Assumptions extended to MLR:

1. Linearity: The relationship between variables is linear.
2. Random Sampling: The data is a representative sample.
3. No Perfect Collinearity: No independent variable is a perfect linear combination of others.
4. Zero Conditional Mean: The error term has an expected value of zero given the \(X\)s. \(E(\epsilon | X) = 0\). (This is the most important one for bias!)
5. Homoskedasticity: The variance of the error term is constant across all observations.

Summary Key Takeaways:
- Coefficients: Interpret them "holding everything else constant."
- Adjusted \(R^2\): Use it to compare models; it penalizes complexity.
- F-test: Tests the whole model; T-test tests one variable.
- Multicollinearity: High correlation between \(X\) variables leads to high standard errors and low t-stats.
- OVB: Occurs when a relevant variable is missing, leading to biased estimates.