Welcome to Extensions of Multiple Regression!
Hello there! If you’ve already tackled basic Multiple Linear Regression (MLR), you know it’s a powerful tool for predicting variables. But the real world is messy. Sometimes our data isn't just numbers—it’s categories (like "Sector" or "Rating"). Sometimes variables interact with each other in strange ways. That is where Extensions of Multiple Regression come in.
Think of this chapter as the "Pro Version" of regression. We are going to learn how to handle more complex scenarios so your models can be more accurate and useful in financial analysis. Don't worry if this seems tricky at first; we will break it down piece by piece!
1. Qualitative Independent Variables (Dummy Variables)
Standard regression uses "quantitative" data (like price or earnings). But what if you want to know if being in the "Technology Sector" affects a stock's return? You can't put the word "Technology" into an equation. Instead, we use Dummy Variables.
What is a Dummy Variable?
A dummy variable is a "binary" variable. It only takes two values: 1 (if the condition is true) or 0 (if it is false). Think of it like an On/Off switch for a specific characteristic.
The "n-1" Rule (The Dummy Variable Trap)
This is a very common exam topic! If you have n categories, you must only include \( n-1 \) dummy variables in your model. For example, if you are modeling the four seasons (Spring, Summer, Fall, Winter), you only need 3 dummy variables.
Why? Because if you know it's not Spring, Summer, or Fall, it must be Winter. Including all 4 causes "perfect multicollinearity," which crashes the model. This is called the Dummy Variable Trap.
Interpreting the Intercept
In a model with dummy variables, the intercept (\( b_0 \)) represents the "base case"—the category that was left out (where all dummies = 0). The coefficient of a dummy variable tells you the difference between that category and the base case.
Quick Review Box:
- Dummy Variable: 1 or 0.
- Number of dummies: Categories minus one.
- Intercept: Represents the excluded category.
2. Interaction Terms
Sometimes, the effect of one variable depends on the value of another variable. For example, the effect of "Years of Experience" on "Salary" might be different for "Men" vs "Women." To capture this, we use an Interaction Term.
How it works
An interaction term is created by multiplying two independent variables together. The model looks like this:
\( Y = b_0 + b_1 X_1 + b_2 X_2 + b_3(X_1 \times X_2) \)
Analogy: Coffee and Sugar
Imagine \( X_1 \) is Coffee and \( X_2 \) is Sugar. \( b_1 \) is how much you like coffee alone. \( b_2 \) is how much you like sugar alone. But the interaction term \( b_3 \) captures the "magic" that happens when you put them together. The effect of sugar depends on whether it's in coffee or just eaten raw!
Key Takeaway: If the coefficient of the interaction term (\( b_3 \)) is statistically significant (p-value < 0.05), it means the relationship between the dependent variable and \( X_1 \) changes depending on the level of \( X_2 \).
3. Qualitative Dependent Variables
So far, we have predicted a number (like stock price). But what if we want to predict a category? For example: Will a company go bankrupt? (Yes/No). Will a bond be upgraded? (Yes/No).
In these cases, our Dependent Variable (Y) is qualitative. We use three main types of models:
A. Probit and Logit Models
These models are used when the outcome is binary (0 or 1). Instead of predicting the exact value of Y, they predict the probability that Y will be 1.
- Logit Model: Based on a "logistic" distribution.
- Probit Model: Based on a "normal" distribution.
Note: For the CFA exam, you don't need to know the complex math behind these, just that they estimate probabilities and ensure the result is always between 0 and 1.
B. Discriminant Analysis
This creates a "score" based on independent variables to sort observations into groups. Think of a Z-score used to predict bankruptcy. If the score is above a certain threshold, the company is "Healthy"; below it, it's "At Risk."
Common Mistake: Don't use a standard Linear Regression (OLS) when the dependent variable is 0/1. It can give you predicted values like 1.5 or -0.2, which make no sense for a Yes/No question!
4. Evaluating Model Specification
Building a model is easy; building a good model is hard. Model Specification refers to choosing the right variables and the right functional form.
Omitted Variable Bias
If you leave out a variable that is important AND correlated with the variables you did include, your model results will be biased and inconsistent. This is the "forgotten ingredient" problem. It makes your remaining coefficients look more (or less) important than they actually are.
Inappropriate Form of Variables
Sometimes the relationship isn't a straight line. We might need to transform the data:
- Log-Log Model: Use when you expect a percentage change in X to lead to a percentage change in Y (constant elasticity).
- Log-Lin Model: Use when Y grows at a constant percentage rate (like compound interest).
- Lin-Log Model: Use when X has a diminishing effect on Y.
5. Dealing with Outliers and Influential Points
One "weird" data point can ruin your entire regression. We need to identify and handle them.
Outliers vs. Influential Points
- Outlier: An observation with a very large residual (it's far away from the regression line).
- High Leverage Point: An observation where the independent variable (X) is an extreme value.
- Influential Point: A point that, if removed, would significantly change the estimated coefficients.
How to detect them?
Cook’s Distance (Cook’s D): This is the most common metric. If Cook's D is large, that specific data point is highly influential and might be distorting your results.
How to fix it?
If you have "dirty" data with many outliers, you can use Robust Regression or M-estimation. These methods reduce the weight given to outliers so they don't "pull" the regression line too far away from the rest of the data.
Summary Checklist:
1. Did I use \( n-1 \) dummy variables?
2. Is there an interaction effect I missed?
3. Is my dependent variable a category? (If yes, use Logit/Probit).
4. Did I check for influential outliers using Cook's D?
Keep going! You're mastering the tools that real-world quantitative analysts use every day. Regression might feel heavy on the math, but it's really just about telling a story with data.