Welcome to Linear Regression!

Hello there! Welcome to one of the most practical and widely used chapters in the CS1 syllabus. If you’ve ever wondered how insurers decide how much to charge for a car insurance policy based on the driver's age, or how economists predict future inflation, you are thinking about Linear Regression.

In this chapter, we aren't just looking at whether two things are related; we are building a mathematical model to describe that relationship. Don't worry if you find statistics a bit daunting—we are going to break this down step-by-step using simple analogies and clear logic. Let’s dive in!

1. What is a Linear Regression Model?

At its heart, simple linear regression is about finding the "best-fit" straight line through a set of data points. We use one variable (the Independent Variable, \( x \)) to predict another variable (the Dependent Variable, \( Y \)).

The Formula

The standard model for a single observation is written as:
\( Y_i = \alpha + \beta x_i + \epsilon_i \)

Let's break down these pieces:

  • \( Y_i \) (The Dependent Variable): This is what we want to predict (e.g., the cost of a claim).
  • \( x_i \) (The Independent Variable/Regressor): This is the information we already have (e.g., the age of the car).
  • \( \alpha \) (The Intercept): This is the value of \( Y \) when \( x \) is zero.
  • \( \beta \) (The Slope): This tells us how much \( Y \) changes for every 1-unit increase in \( x \).
  • \( \epsilon_i \) (The Error Term): This is the "random noise." In the real world, data points never fall perfectly on a line. The error term accounts for the distance between the actual point and the line.

Analogy: Imagine you are baking a cake. The height of the cake (\( Y \)) depends on the amount of baking powder you use (\( x \)). The intercept (\( \alpha \)) is the height of the batter before it rises, the slope (\( \beta \)) is how much it rises per gram of powder, and the error (\( \epsilon \)) represents random factors like the oven temperature being slightly off.

Quick Review: The goal of regression is to estimate the values of \( \alpha \) and \( \beta \) so we can make accurate predictions.

2. The "Must-Have" Assumptions (LINE)

For our linear regression model to be valid and "statistically happy," we make four big assumptions. You can remember them with the mnemonic LINE:

  • L - Linearity: The relationship between \( x \) and \( Y \) must be a straight line.
  • I - Independence: The error terms (\( \epsilon_i \)) must be independent of each other. One person's claim shouldn't affect another's.
  • N - Normality: We assume the errors follow a Normal Distribution with a mean of zero: \( \epsilon_i \sim N(0, \sigma^2) \).
  • E - Equal Variance (Homoscedasticity): The "spread" of the errors should be constant across all values of \( x \).

Common Mistake: Students often forget that the normality assumption applies to the errors (\( \epsilon \)), not necessarily the raw \( x \) or \( Y \) values themselves!

3. Estimating the Parameters: Ordinary Least Squares (OLS)

How do we actually find the "best" line? We use a method called Ordinary Least Squares (OLS).

The "best" line is the one that makes the Sum of Squared Errors (SSE) as small as possible. We square the errors because we want to treat positive and negative distances the same way and penalize large misses more heavily.

The Magic Formulas

Using calculus, we derive the estimators for our slope and intercept:

The Slope (\( \hat{\beta} \)):
\( \hat{\beta} = \frac{S_{xy}}{S_{xx}} \)
where \( S_{xy} = \sum (x_i - \bar{x})(y_i - \bar{y}) \) and \( S_{xx} = \sum (x_i - \bar{x})^2 \).

The Intercept (\( \hat{\alpha} \)):
\( \hat{\alpha} = \bar{y} - \hat{\beta}\bar{x} \)

Did you know? The regression line always passes through the point of the means \( (\bar{x}, \bar{y}) \). If you calculate a line that doesn't go through the average of your data, double-check your math!

4. How Good is Our Model? (Goodness of Fit)

Once we have a line, we need to know if it's actually doing a good job. We use the Coefficient of Determination, known as \( R^2 \).

\( R^2 = \frac{SSR}{SST} \)

  • SST (Total Sum of Squares): The total variation in our data.
  • SSR (Regression Sum of Squares): The variation explained by our model.
  • SSE (Error Sum of Squares): The variation our model couldn't explain (the "leftovers").

Interpretation: \( R^2 \) is always between 0 and 1.
- \( R^2 = 1 \): Perfect fit (all points are on the line).
- \( R^2 = 0 \): The model explains nothing.

Key Takeaway: An \( R^2 \) of 0.85 means that 85% of the variation in \( Y \) is explained by the relationship with \( x \). The other 15% is just random noise or other factors.

5. Statistical Inference: Is the Slope Significant?

In CS1 exams, you are often asked: "Is there actually a relationship, or is this just a coincidence?"

We test this using a Hypothesis Test on the slope \( \beta \).

  • Null Hypothesis (\( H_0 \)): \( \beta = 0 \) (There is no relationship).
  • Alternative Hypothesis (\( H_1 \)): \( \beta \neq 0 \) (There is a relationship).

We use a t-test because we usually don't know the true variance (\( \sigma^2 \)) and have to estimate it from our data.

The Test Statistic:
\( t = \frac{\hat{\beta} - 0}{SE(\hat{\beta})} \)
where \( SE(\hat{\beta}) \) is the Standard Error of the slope. We compare this to a t-distribution with \( n - 2 \) degrees of freedom.

Don't worry: Most exam questions will give you the Standard Error or enough data to find it using \( \hat{\sigma}^2 = \frac{SSE}{n-2} \).

6. Making Predictions

The ultimate goal of regression for an actuary is prediction. There are two types of predictions:

  1. Predicting the Mean Response: Predicting the average \( Y \) for a given \( x \). (e.g., "What is the average claim for all 30-year-olds?")
  2. Predicting an Individual Value: Predicting the specific \( Y \) for one specific case. (e.g., "What is the claim for this specific 30-year-old named John?")

Important Note: Predicting for an individual (John) is always less certain than predicting an average. Therefore, the Prediction Interval (for individuals) is always wider than the Confidence Interval (for the mean).

Summary and Tips for Success

Key Points Recap:

  • The model is \( Y = \alpha + \beta x + \epsilon \).
  • Use OLS to find \( \hat{\alpha} \) and \( \hat{\beta} \) by minimizing squared errors.
  • The LINE assumptions are vital for the model's validity.
  • \( R^2 \) tells you the percentage of variation explained.
  • Use a t-test to check if the slope is significantly different from zero.

Exam Tip: When calculating \( S_{xx} \) and \( S_{xy} \), keep your intermediate decimals! Rounding too early can lead to a significantly different final answer, which might cost you marks in the CS1 exam.

You’ve got this! Regression might look like a lot of formulas, but it’s really just the science of drawing the best possible line through a cloud of data. Keep practicing the calculations, and the logic will follow!