Welcome to Least Squares Regression and Residuals!

In our previous look at Data Analysis, we explored how to see if two variables were related using correlation. Now, we are going to take it a step further. Least Squares Regression is all about finding the "perfect" straight line that fits our data. This line allows us to make predictions and understand the specific relationship between an independent variable (like time spent studying) and a dependent variable (like exam scores).

Don't worry if the math looks intimidating at first! For this course, your calculator does the heavy lifting for calculations, while you focus on interpreting what the numbers actually mean in the real world.

1. The Regression Line Equation

The least squares regression line is written as a straight-line equation:

\(y = a + bx\)

Here is what those letters represent:

  • \(y\): The dependent (or response) variable. This is what we are trying to predict.
  • \(x\): The independent (or explanatory) variable. This is the variable we think "explains" the change in \(y\).
  • \(b\): The gradient (slope). This tells us how much \(y\) changes for every 1-unit increase in \(x\).
  • \(a\): The y-intercept. This is the predicted value of \(y\) when \(x = 0\).

How do we find \(a\) and \(b\)?

In your exam, you should use your calculator's Statistics mode to get these values directly from raw data. However, the Statistical Formulae and Tables booklet provides these formulas if you ever need to calculate them from summary statistics:

\(b = \frac{S_{xy}}{S_{xx}}\)

\(a = \bar{y} - b\bar{x}\)

(Where \(\bar{x}\) and \(\bar{y}\) are the means of your data sets.)

Quick Tip: Context is King!

If you calculate that \(b = 2.5\) in a study about "Hours of sunshine" (\(x\)) and "Ice cream sales" (\(y\)), you would interpret it as: "For every extra hour of sunshine, we predict ice cream sales will increase by 2.5 units."

2. Making Predictions: Interpolation vs. Extrapolation

Once you have your equation \(y = a + bx\), you can plug in a value of \(x\) to predict a value for \(y\).

Interpolation (The Safe Zone)

Interpolation is making a prediction inside the range of the data you already have. For example, if you measured heights of children aged 5 to 10, predicting the height of a 7-year-old is interpolation. Reliability: Generally very reliable because we have evidence of how the data behaves in this area.

Extrapolation (The Danger Zone)

Extrapolation is making a prediction outside the range of your data. If you used your "ages 5 to 10" data to predict the height of a 40-year-old, the model would likely suggest they are 12 feet tall! Reliability: Very risky and often unreliable. We cannot assume the linear trend continues forever.

Key Takeaway: Always be skeptical of predictions made through extrapolation.

3. Understanding Residuals

No model is perfect. A residual is the vertical distance between an actual observed data point and the line our model predicted. Think of it as the "error" of our prediction.

The Residual Formula

\(\text{Residual} = \text{Observed } y - \text{Predicted } y\)

In symbol form: \(e_i = y_i - (a + bx_i)\)

  • If the residual is positive, the actual data point is above the regression line (the model under-predicted).
  • If the residual is negative, the actual data point is below the regression line (the model over-predicted).
  • If the residual is zero, the point lies exactly on the line.
Example Step-by-Step:

Suppose your regression line is \(y = 10 + 2x\). You have a data point where \(x = 5\) and the actual \(y = 22\).

  1. Calculate the predicted value: \(y = 10 + 2(5) = 20\).
  2. Calculate the residual: \(22 - 20 = +2\).
  3. The model was 2 units too low for this specific point.

4. Using Residuals to Evaluate the Model

Residuals aren't just "mistakes"—they are valuable tools for checking if our linear model is actually a good fit for the data.

Identifying Outliers

An outlier in regression is a point with a very large residual (it is much further from the line than the other points). You can spot these by looking at a scatter graph with the line drawn on it or by looking for unusually high numerical residual values.

Checking for "Goodness of Fit"

When you plot the residuals on a separate graph (a residual plot), you are looking for randomness:

  • Good Model: The residuals are randomly scattered above and below the zero-line with no obvious pattern. This suggests a linear model is appropriate.
  • Bad Model: If the residuals form a curve (like a "U" shape), it suggests the original data wasn't actually linear, and you should perhaps use a different type of model.

Did you know? The "Least Squares" method gets its name because the line is chosen to make the sum of the squares of all the residuals as small as possible!

Quick Review: Common Pitfalls to Avoid

  • Don't swap \(x\) and \(y\): Always ensure the explanatory variable is \(x\) and the response variable is \(y\) before typing numbers into your calculator.
  • Contextual Interpretation: When asked to interpret the gradient \(b\), always mention the specific units from the question (e.g., kilograms, meters, pounds).
  • Extrapolation Warnings: If an exam question asks "How reliable is this prediction?", and the \(x\) value is outside the data range, your answer should almost always be "Not reliable due to extrapolation."
  • Residual Calculation: Remember it is \(Actual - Predicted\). If you do \(Predicted - Actual\), your signs will be backward!

This concludes the notes for Least Squares Regression and Residuals. For information on how to test the strength of these relationships, refer to the chapter on "Correlation: Spearman and Pearson".