Linear Regression

Welcome to the World of Linear Regression!

Hi there, future FRM holder! Don't let the name "Linear Regression" intimidate you. At its heart, this chapter is about one simple thing: finding relationships. In the world of finance, we are always trying to figure out how one thing (like interest rates) affects another (like bond prices). Linear regression is the mathematical tool that helps us draw a "line of best fit" through messy data to see what’s really going on. If you've ever looked at a scatterplot and tried to draw a straight line through the middle of the dots, you’ve already done the basic work of a regression!

By the end of these notes, you’ll understand how to build these models, what the "Beta" coefficients actually mean, and how to tell if your model is a reliable tool or just a lucky guess.

1. The Simple Linear Regression Model

In a simple linear regression, we look at the relationship between two variables: the Dependent Variable (\(Y\)) and the Independent Variable (\(X\)).

The Equation:
\( Y_i = \beta_0 + \beta_1 X_i + \epsilon_i \)

Let's break this down into digestible parts:
• \(Y_i\) (Dependent Variable): This is what you are trying to predict or explain. (e.g., a stock’s return).
• \(X_i\) (Independent/Explanatory Variable): This is the factor you think influences \(Y\). (e.g., the market return).
• \(\beta_0\) (Intercept): This is the value of \(Y\) if \(X\) is zero. On a graph, it's where the line crosses the vertical axis.
• \(\beta_1\) (Slope Coefficient): This is the most important part! it tells us how much \(Y\) changes for every 1-unit change in \(X\).
• \(\epsilon_i\) (Error Term): This is the "noise." Real-world data is never perfect. The error term represents everything else that affects \(Y\) that we didn't include in our model.

Real-World Example: Think of your exam score (\(Y\)) based on hours studied (\(X\)). The \(\beta_1\) would tell you how many extra points you get for every hour you study. The \(\epsilon\) would be things like how much sleep you got or how loud the exam room was.

Key Takeaway: Regression helps us quantify exactly how much one variable moves when another one changes.

2. Ordinary Least Squares (OLS) Estimation

How do we actually find the "best" line? We use a method called Ordinary Least Squares (OLS). Don't worry if this seems tricky at first; the concept is actually quite intuitive.

OLS tries to find the line that makes the sum of the squared vertical distances between the actual data points and the line as small as possible. We square the distances because we don't want positive and negative errors to cancel each other out—we want to penalize all "misses."

The Formulas (for your reference):
The estimated slope (\(\hat{\beta}_1\)) is calculated as:
\( \hat{\beta}_1 = \frac{Cov(X, Y)}{Var(X)} \)
The intercept (\(\hat{\beta}_0\)) is calculated as:
\( \hat{\beta}_0 = \bar{Y} - \hat{\beta}_1 \bar{X} \)

Quick Review: OLS gives us the "Best Fit" line by minimizing the squared errors. It always passes through the mean of \(X\) and the mean of \(Y\).

3. The Golden Rules: Assumptions of OLS

For OLS to give us reliable results (the "Best Linear Unbiased Estimator" or BLUE), certain rules must be followed. If these rules are broken, our model might lie to us!

1. Linearity: The relationship between \(X\) and \(Y\) must be a straight line.
2. Random Sampling: The data must be representative of the population.
3. No Perfect Collinearity: \(X\) must actually vary. If \(X\) is always the same number, we can't see how it affects \(Y\).
4. Exogeneity: The error term must not be related to \(X\). This means \(E(\epsilon | X) = 0\). If this is violated, we have Omitted Variable Bias.
5. Homoskedasticity: The "spread" or variance of the error terms should be constant across all values of \(X\).

Did you know? If these assumptions hold, the OLS estimators are the most efficient ones available. We call this the Gauss-Markov Theorem.

4. Goodness of Fit: How Good is Our Model?

Once we have a line, we need to know if it's actually doing a good job. We use Coefficient of Determination (\(R^2\)) to measure this.

\(R^2\) Explained:
\(R^2\) tells us the percentage of the variation in \(Y\) that is explained by \(X\).
• An \(R^2\) of 0.80 means 80% of the movement in \(Y\) is explained by our model.
• \(R^2\) always ranges from 0 to 1.

Formula:
\( R^2 = \frac{Explained Sum of Squares (ESS)}{Total Sum of Squares (TSS)} \)
Or: \( 1 - \frac{Sum of Squared Residuals (SSR)}{TSS} \)

Memory Aid: Think of TSS as a "Total Pizza." ESS is the part of the pizza you actually ate (explained), and SSR is the crust left behind (unexplained error).

5. Hypothesis Testing: Is the Relationship Real?

Just because we found a slope (\(\beta_1\)) doesn't mean it's "real." It could just be random luck in our small sample. We use a t-test to check.

Step-by-Step Testing:
1. Set the Null Hypothesis (\(H_0\)): Usually, we assume \(\beta_1 = 0\) (meaning \(X\) has no effect on \(Y\)).
2. Calculate the t-statistic:
\( t = \frac{\hat{\beta}_1 - Hypothesized Value}{Standard Error of \hat{\beta}_1} \)
3. Compare to Critical Value: If the absolute value of your t-stat is larger than the critical value (usually around 2.0 for a 5% significance level), you Reject the Null!

Key Term: The p-value is the probability that we would see this result if the Null Hypothesis were actually true. A low p-value (typically < 0.05) means the relationship is "statistically significant."

Common Mistake to Avoid: Don't confuse Statistical Significance with Economic Significance. A variable might be statistically significant (p < 0.05) but have such a tiny slope (\(\beta_1\)) that it doesn't actually matter in the real world of finance.

6. Confidence Intervals

Instead of just giving one number for our estimate, we often provide a range. This is the Confidence Interval.

The Formula:
\( \hat{\beta}_1 \pm (t_{critical} \times SE_{\hat{\beta}_1}) \)

If we want a 95% confidence interval, we are saying we are 95% sure the true population parameter falls within this range. If the range includes zero, then the relationship is not statistically significant!

Quick Summary Table:
• High t-stat (> 2): Significant relationship.
• Low p-value (< 0.05): Significant relationship.
• CI includes 0: Not significant.

Final Wrap-Up

Linear regression is the backbone of quantitative finance. Whether you are calculating the "Beta" of a stock or trying to predict inflation, you are using these tools. Remember: OLS minimizes squared errors, \(R^2\) measures fit, and the t-test checks for significance. Keep these three pillars in mind, and you'll do great on this section of the FRM exam!