Welcome to Simple Linear Regression!
In your CFA journey so far, you’ve learned how to calculate averages and describe data. But what if you want to use one piece of information to predict another? For example, can we predict a stock’s return based on how the overall market is performing? That is exactly what Simple Linear Regression (SLR) allows us to do. Think of it as finding the "best-fit" line through a scatterplot of data points to help us understand the relationship between two variables.
Don't worry if the math looks intimidating at first—we are going to break it down step-by-step!
1. The Basics: What are we building?
In a simple linear regression, we are looking at the relationship between two variables:
1. The Dependent Variable (Y): This is what we are trying to explain or predict. It is also called the "Response" or "Explained" variable.
2. The Independent Variable (X): This is the factor we use to explain Y. It is also called the "Explanatory" or "Predictor" variable.
The Regression Equation:
\( Y_i = b_0 + b_1X_i + \epsilon_i \)
Where:
- \( Y_i \) is the dependent variable.
- \( b_0 \) is the Intercept (where the line hits the vertical axis).
- \( b_1 \) is the Slope (how much Y changes for every 1-unit change in X).
- \( \epsilon_i \) is the Error Term (the difference between the actual data point and the line).
Real-World Analogy: Imagine you are predicting your electricity bill (Y) based on the outside temperature (X). The Intercept (\( b_0 \)) is the basic service fee you pay even if you use no power. The Slope (\( b_1 \)) is how much your bill goes up for every extra degree of heat.
Quick Takeaway:
Dependent (Y) = What we want to know.
Independent (X) = What we already know.
Slope (\( b_1 \)) = The "strength" and "direction" of the relationship.
2. The "Best" Line: Ordinary Least Squares (OLS)
How do we decide which line is the "best" fit? We use a method called Ordinary Least Squares (OLS).
The OLS method chooses the line that minimizes the sum of the squared vertical distances between the actual data points and the regression line. We square the distances so that "misses" above the line don't cancel out "misses" below the line.
Did you know? We call the vertical distance between an actual point and the predicted point on the line a Residual (\( \hat{\epsilon} \)).
3. Assumptions of Simple Linear Regression
For our regression model to be reliable, we have to assume a few things are true about the data. A popular way to remember these is the "LINE" mnemonic:
1. Linearity: The relationship between X and Y is a straight line.
2. Independence: The observations (errors) are independent of each other.
3. Normality: The error terms are normally distributed.
4. Equal Variance: The variance of the error terms is constant across all levels of X (this is a fancy term called Homoscedasticity).
Common Pitfall: Students often forget "Homoscedasticity." Just remember: "Homo" means same, and "scedasticity" refers to spread. We want the "same spread" of errors all along the line!
4. Evaluating the Model: How good is our "fit"?
Once we have a line, we need to know if it's actually useful. We use three main tools:
A. Coefficient of Determination (\( R^2 \))
\( R^2 \) tells us the percentage of the variation in Y that is explained by X.
- It ranges from 0 to 1.
- An \( R^2 \) of 0.80 means 80% of the movement in Y is explained by X.
- In simple linear regression, \( R^2 \) is simply the correlation coefficient (\( r \)) squared.
B. Standard Error of Estimate (SEE)
The \( SEE \) measures the average "distance" that the data points fall from the regression line.
- Lower is better! A smaller \( SEE \) means the data points are very close to the line, making our predictions more accurate.
C. ANOVA (Analysis of Variance)
ANOVA breaks down the total variation in the dependent variable into two parts:
1. SSR (Sum of Squares Regression): The variation explained by our model.
2. SSE (Sum of Squares Error): The variation our model failed to explain (the "leftovers").
3. SST (Total Sum of Squares): The total variation (\( SST = SSR + SSE \)).
The "Pizza" Analogy: Imagine SST is a whole pizza. SSR is the amount of pizza your friend ate (explained). SSE is the amount of pizza dropped on the floor (unexplained/error). The \( R^2 \) is just the percentage of the pizza your friend successfully ate (\( SSR / SST \)).
5. Hypothesis Testing: Is the Slope Significant?
Just because our calculation gives us a slope (\( b_1 \)) doesn't mean a real relationship exists. It could be due to random chance! We test this using a t-test.
The Null Hypothesis (\( H_0 \)): \( b_1 = 0 \) (There is NO relationship).
The Alternative Hypothesis (\( H_a \)): \( b_1 \neq 0 \) (There IS a significant relationship).
To test this, we calculate:
\( t = \frac{\hat{b}_1 - 0}{s_{b1}} \)
Where \( s_{b1} \) is the standard error of the slope coefficient.
If the calculated t-statistic is greater than the critical t-value, we reject the null and say the relationship is statistically significant!
6. Making a Prediction
This is the fun part! Once you have your coefficients (\( b_0 \) and \( b_1 \)), you can predict Y for any value of X.
Step-by-Step Prediction:
1. Identify the given intercept (\( b_0 \)) and slope (\( b_1 \)).
2. Plug in the given value of \( X \).
3. Calculate \( \hat{Y} = b_0 + b_1(X) \).
Example: If \( b_0 = 2 \), \( b_1 = 0.5 \), and we want to predict Y when X is 10:
\( \hat{Y} = 2 + 0.5(10) = 2 + 5 = 7 \).
7. Limitations of Regression
Before you go, remember that regression isn't magic. It has limits:
- Outliers: One extreme data point can "pull" the line and ruin the model.
- Correlation vs. Causation: Just because X and Y move together doesn't mean X causes Y. (Example: Ice cream sales and shark attacks both go up in summer, but ice cream doesn't cause shark attacks!)
- Non-linear relationships: If the real relationship is a curve, a straight line will be a poor fit.
Quick Review Box:
OLS: Minimizes the sum of squared residuals.
\( R^2 \): The "scorecard" (higher = better fit).
Standard Error (SEE): The "mistake" measure (lower = better).
t-test: Used to see if the slope is significantly different from zero.
Keep practicing! Simple Linear Regression is the foundation for almost all advanced modeling you will see in Level II and Level III. You've got this!