p-values, Type I and Type II errors

Introduction: Making Decisions with Data

Welcome to one of the most important parts of AP Statistics! In previous chapters, you learned how to set up hypotheses and calculate test statistics. Now, we are getting to the "so what?" of the process. How do we actually decide if our data provides enough evidence to support a claim? And what happens if we make the wrong call? Don't worry if this feels a bit like a logic puzzle at first—we will break it down using simple steps and real-world analogies.

In this chapter, we focus on p-values, Type I errors, and Type II errors within the context of Unit 3: Inference for Categorical Data (Proportions).

Section 1: Understanding p-Values

The p-value is the heart of a significance test. It tells us how "surprising" our sample data is if we assume the null hypothesis (\(H_0\)) is actually true.

What is a p-Value?

A p-value is the probability of getting a sample proportion (\(\hat{p}\)) as extreme as, or more extreme than, the one observed in our study, given that the null hypothesis is true.

The "Surprise" Analogy: Imagine a friend claims they can tell the difference between bottled water and tap water (\(H_a\)). The null hypothesis (\(H_0\)) is that they are just guessing (\(p = 0.5\)). If they get \(10\) out of \(10\) right, that is very surprising! The p-value would be very small, meaning it’s unlikely they got those results just by luck.

Interpreting p-Values in Context

When you write an interpretation on the AP Exam, you must use specific language. You should say: "Assuming the true population proportion of [context] is [value in \(H_0\)], there is a [p-value] probability of getting a sample proportion as extreme as or more extreme than the one observed."

Quick Review Box:
- Small p-value: The data is surprising. We have evidence against \(H_0\).
- Large p-value: The data is NOT surprising. It is consistent with \(H_0\).

Section 2: Making the Decision

To make a final decision, we compare the p-value to a pre-set threshold called the significance level, denoted by the Greek letter alpha (\(\alpha\)). Common values for \(\alpha\) are \(0.05\), \(0.01\), or \(0.10\).

The Decision Rules:

1. If p-value \( < \alpha\): We reject \(H_0\). We have convincing evidence for the alternative hypothesis (\(H_a\)).
2. If p-value \(\ge \alpha\): We fail to reject \(H_0\). We do NOT have convincing evidence for the alternative hypothesis (\(H_a\)).

CRITICAL RULE: Never say you "accept" the null hypothesis. In statistics, we either find enough evidence to reject it, or we don't. Think of a jury: they find a defendant "not guilty," which is not the same as proving they are innocent!

Section 3: Type I and Type II Errors

Because we are using samples to make guesses about whole populations, there is always a chance we make the wrong decision. We categorize these mistakes into two types.

Type I Error: The "False Alarm"

A Type I Error occurs if we reject the null hypothesis (\(H_0\)) when, in reality, the null hypothesis is actually true.

Example: A medical test says a patient has a disease (\(H_a\)), but the patient is actually healthy (\(H_0\)).
Memory Trick: Type I is when we say something Is happening, but it isn't.

Type II Error: The "Missed Opportunity"

A Type II Error occurs if we fail to reject the null hypothesis (\(H_0\)) when, in reality, the alternative hypothesis (\(H_a\)) is true.

Example: A medical test says a patient is healthy (\(H_0\)), but the patient actually has the disease (\(H_a\)).
Memory Trick: Type II is when we are IIgnorant of the truth.

Summary Table of Decisions

If \(H_0\) is True: Rejecting \(H_0\) is a Type I Error.
If \(H_a\) is True: Failing to reject \(H_0\) is a Type II Error.

Section 4: Consequences and Relationships

On the AP Exam, you will often be asked to describe the consequences of these errors in a specific context. This means explaining what actually happens in real life if the error is made.

Example Context: Testing a new, expensive fertilizer to see if it increases the proportion of plants that bloom (\(p\)).
- Type I Consequence: We think the fertilizer works and spend lots of money buying it, but it doesn't actually help.
- Type II Consequence: We think the fertilizer doesn't work and don't use it, missing out on a way to get more flowers.

The Relationship Between \(\alpha\), Type I, and Type II

There is a "tug-of-war" between these two errors:

1. The probability of making a Type I Error is exactly equal to the significance level (\(\alpha\)). If you set \(\alpha = 0.05\), you have a \(5\%\) chance of a Type I error.
2. If you decrease \(\alpha\) (e.g., from \(0.05\) to \(0.01\)), you make it harder to reject \(H_0\). This decreases the risk of a Type I error but increases the risk of a Type II error.
3. If you increase \(\alpha\), the risk of a Type I error increases, but the risk of a Type II error decreases.

Key Takeaway: You can't eliminate error entirely, but you can choose which error is "worse" for your specific situation and adjust your significance level accordingly!

Common Mistakes to Avoid

1. Using "it": Avoid saying "It is significant." Instead, be specific: "The sample proportion of plants that bloomed is statistically significant at the \(\alpha = 0.05\) level."
2. Misinterpreting p-value: Never say the p-value is "the probability that the null hypothesis is true." The p-value is a probability about the data, not the hypothesis.
3. Forgetting Context: When describing errors, always mention the specific real-world subjects (patients, voters, products, etc.) mentioned in the prompt.

Next Chapters in Unit 3: You will use these concepts to carry out full significance tests for one proportion (\(z\)-tests) and compare two proportions.