One-sample, two-sample, paired and permutation tests

Introduction: Making Sense of Data

Welcome! In the world of actuarial science, we are often detectives. We look at data and ask: "Is this result just a lucky coincidence, or is there a real trend here?" For example, if a new health initiative seems to reduce insurance claims, how do we prove it’s not just random chance?

This chapter focuses on Hypothesis Testing for specific scenarios. We will look at how to test one group, compare two independent groups, handle "linked" data (paired), and use a clever "shuffling" method called a permutation test when we don't want to make assumptions about the data's distribution.

Note: This chapter assumes you are familiar with basic concepts like the Null Hypothesis (\(H_0\)) and p-values. If you need a refresher, check out the chapter on "Hypothesis testing concepts, errors and power".

1. One-Sample Tests: Checking a Single Group

A one-sample test is used when you want to compare a sample from a single population against a known value (a "benchmark").

Testing the Mean (\(\mu\)) of a Normal Distribution

If we assume our data follows a Normal distribution, we usually want to test the mean. There are two main paths:

• Case A: Variance (\(\sigma^2\)) is known. We use the Z-test. The test statistic is:
  \(Z = \frac{\bar{X} - \mu_0}{\sigma / \sqrt{n}} \sim N(0, 1)\)
• Case B: Variance (\(\sigma^2\)) is unknown. This is much more common in real life! We use the t-test. We replace the population \(\sigma\) with the sample standard deviation \(s\). The test statistic is:
  \(t = \frac{\bar{X} - \mu_0}{s / \sqrt{n}} \sim t_{n-1}\)

Testing the Variance (\(\sigma^2\)) of a Normal Distribution

Sometimes actuaries care more about risk (volatility) than the average. To test if the variance equals a specific value (\(\sigma^2_0\)), we use the Chi-square test:
\(\chi^2 = \frac{(n-1)S^2}{\sigma^2_0} \sim \chi^2_{n-1}\)

Testing Proportions (Binomial) and Rates (Poisson)

For Binomial (\(p\)) and Poisson (\(\lambda\)) data, we usually rely on the Normal Approximation if the sample size is large enough.
For Binomial: \(Z = \frac{\hat{p} - p_0}{\sqrt{\frac{p_0(1-p_0)}{n}}} \sim N(0, 1)\)
For Poisson: \(Z = \frac{\bar{X} - \lambda_0}{\sqrt{\lambda_0 / n}} \sim N(0, 1)\)

Quick Review: Always check if you know the variance. If you don't, and the data is Normal, reach for the \(t\)-distribution!

2. Two-Sample Tests: Comparing Two Independent Groups

Actuaries often compare two different groups, such as the claim amounts of Policyholder Group A versus Group B.

Comparing Two Normal Means

To test if \(\mu_1 = \mu_2\), we look at the difference \(\bar{X}_1 - \bar{X}_2\). In the CS1 syllabus, we focus on the "basic" case where we assume the two populations have the same unknown variance (\(\sigma^2_1 = \sigma^2_2\)). We calculate a pooled sample variance (\(s_p^2\)):
\(s_p^2 = \frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1 + n_2 - 2}\)

The test statistic is:
\(t = \frac{(\bar{X}_1 - \bar{X}_2) - 0}{s_p \sqrt{\frac{1}{n_1} + \frac{1}{n_2}}} \sim t_{n_1+n_2-2}\)

Comparing Two Normal Variances

To see if one group is more volatile than another, we use the F-test. We divide the larger sample variance by the smaller one:
\(F = \frac{s_1^2}{s_2^2} \sim F_{n_1-1, n_2-1}\)

Two-Sample Binomial and Poisson

When comparing two proportions (\(p_1\) and \(p_2\)) or two Poisson rates (\(\lambda_1\) and \(\lambda_2\)), we again use the Normal approximation for large samples. We typically use a "pooled" estimate for the standard error under the null hypothesis that the two groups are identical.

Key Takeaway: For two-sample tests, the groups must be independent. If the same people are in both groups, you need the next section!

3. Paired Data: Before and After

What if you measure the blood pressure of 10 people before a medical treatment and then measure the same 10 people after? These are not independent samples; they are paired.

The Trick: Don't treat them as two samples. Instead, calculate the difference for each pair:
\(D_i = X_i - Y_i\)

Now, you simply perform a one-sample t-test on these differences (\(D\)). Your null hypothesis is usually \(H_0: \mu_D = 0\).
\(t = \frac{\bar{D} - 0}{s_D / \sqrt{n}} \sim t_{n-1}\)

Common Mistake: Using a two-sample test for paired data. This is a classic exam trap! If the data points are "matched" or represent the same subject at different times, always use the paired approach.

4. Permutation Tests: The Non-Parametric Approach

All the tests above assume a specific distribution (like Normal). But what if the data looks weird, or the sample size is too small for the Central Limit Theorem to kick in? We use a Permutation Test.

The Concept: "Shuffling the Deck"

Imagine you have two groups, A and B, and you observe a difference in their means. A permutation test asks: "If the group labels (A or B) didn't actually matter, how likely is it that we'd see a difference this big just by randomly re-assigning the labels?"

Step-by-Step Process:

1. Calculate the observed test statistic (e.g., the difference in means between Group A and B).
2. Combine all the data from both groups into one big "pool."
3. Permute: Randomly shuffle the data and split it into two new groups of the same original sizes.
4. Calculate the difference in means for this shuffled version.
5. Repeat this "shuffling" thousands of times to create a distribution of differences.
6. The p-value is the proportion of shuffled differences that are as extreme (or more extreme) than your original observed difference.

Did you know? Permutation tests are incredibly powerful because they make no assumptions about the "shape" of your data. They are often performed using R in Paper B.

Summary Checklist for the Exam

• Identify the number of samples: Is it one group compared to a value, two independent groups, or paired data?
• Check the distribution: If it's Normal and the variance is unknown, use the \(t\)-test.
• Large samples? For Binomial and Poisson, use the Normal approximation.
• Are the groups linked? If yes, use the differences (\(D_i\)) and a paired \(t\)-test.
• No distribution specified? Consider the permutation approach (shuffling labels).

Top Tip: In Paper A, you'll likely do the calculations by hand using tables. In Paper B, you will use R functions like t.test(). Make sure you can interpret the output of both!