Asymptotic distribution of MLEs and the bootstrap

Introduction to Asymptotic Distributions and the Bootstrap

Welcome to one of the most powerful chapters in Statistical Inference! Up until now, you have learned how to find a "best guess" for a parameter using Maximum Likelihood Estimation (MLE). But a single number (a point estimate) doesn't tell the whole story. We need to know how reliable that guess is.

In this chapter, we explore two ways to understand the behavior of our estimators:
1. Asymptotic Theory: What happens to our MLE when our sample size \( n \) becomes very large? (Hint: It starts behaving very predictably!)
2. The Bootstrap: What do we do when the math is too hard or we don't have enough data to use standard formulas? We "pull ourselves up by our bootstraps" using modern computing power.

Don't worry if these terms sound intimidating. We will break them down into simple, step-by-step concepts that are easy to apply in your CS1 exams.

1. Asymptotic Distribution of MLEs

The word asymptotic simply refers to what happens as the sample size \( n \) approaches infinity (\( n \to \infty \)). In the actuarial world, we often deal with large datasets, so these "large sample" properties are incredibly useful.

The Main Result

Under certain regularity conditions, as the sample size \( n \) gets larger, the distribution of the MLE, \( \hat{\theta} \), approaches a Normal Distribution. Specifically:

\( \hat{\theta} \sim N(\theta, \frac{1}{I(\theta)}) \)

Where:
\( \theta \) is the true value of the parameter.
\( I(\theta) \) is the Fisher Information.

What is Fisher Information?

Think of Fisher Information as a measure of how much "information" the data carries about the unknown parameter. The more information we have, the smaller the variance of our estimator.
The Fisher Information for a sample of size \( n \) is calculated as:
\( I(\theta) = n \cdot i(\theta) \)
Where \( i(\theta) \) is the information in a single observation:
\( i(\theta) = -E[\frac{d^2}{d\theta^2} \ln f(X; \theta)] \)

Key Takeaways for Asymptotics

1. Asymptotic Unbiasedness: Even if an MLE is biased for small samples, it becomes unbiased as \( n \) grows.
2. Efficiency: The variance of the MLE \( \frac{1}{I(\theta)} \) is the Cramer-Rao Lower Bound. This means that for large samples, no other unbiased estimator can be more precise than the MLE.
3. Normal Approximation: This allows us to use standard normal \( Z \)-tables to calculate probabilities and confidence intervals for MLEs, even if the original data wasn't normal!

Quick Review: As the sample size increases, the MLE becomes more accurate (mean moves toward the truth) and more precise (variance shrinks).

2. The Bootstrap Method

Sometimes, we don't have a large enough sample for asymptotic theory to work, or the formula for the Fisher Information is too complicated to solve. This is where the Bootstrap comes in.

The Concept: Resampling

The Bootstrap is a resampling technique. Imagine you have a small bag of marbles (your original sample). You want to know the variety of marbles in the "big factory" (the population), but you only have this one bag.
To "bootstrap," you take a marble out of your bag, write down its color, and put it back. You repeat this until you have a "new" sample of the same size. This is called sampling with replacement.

Step-by-Step Bootstrap Process

Step 1: Start with your original sample of size \( n \).
Step 2: Draw a new sample (a "bootstrap sample") of size \( n \) from your original data with replacement.
Step 3: Calculate your estimator (e.g., the mean or the MLE) for this bootstrap sample. Let's call this \( \hat{\theta}^* \).
Step 4: Repeat Steps 2 and 3 many times (e.g., 1,000 or 10,000 times) to get a large collection of bootstrap estimates: \( \hat{\theta}^*_1, \hat{\theta}^*_2, ..., \hat{\theta}^*_B \).
Step 5: Use this collection to estimate the properties of your original estimator.

Why Resample "With Replacement"?

If you sampled without replacement, you would just get the exact same data points in a different order every time. By sampling with replacement, you allow some data points to appear twice or more, and some not at all. This mimics the natural variation we would see if we were able to take many different samples from the real population.

Did you know? The term "bootstrap" comes from the phrase "to pull oneself up by one's bootstraps," which implies performing an impossible task without outside help. In statistics, it refers to using the data itself to tell us about its own uncertainty!

3. Estimating Properties Using the Bootstrap

The syllabus requires you to know how to use these bootstrap samples to estimate properties of an estimator, specifically its Bias and Standard Error.

Bootstrap Estimate of Standard Error

Simply calculate the standard deviation of all your bootstrap estimates \( \hat{\theta}^*_1, ..., \hat{\theta}^*_B \). This gives you an estimate of the precision of your estimator.

Bootstrap Estimate of Bias

Bias is the difference between the expected value of an estimator and the true value. In bootstrap terms:
\( \text{Estimated Bias} = (\text{Mean of all } \hat{\theta}^*) - \hat{\theta}_{original} \)
Where \( \hat{\theta}_{original} \) is the estimate calculated from your initial real-world data.

4. Bootstrap Confidence Intervals

One of the most practical uses of the bootstrap is creating confidence intervals (CIs) when we don't want to assume the data follows a specific distribution (like the Normal or Poisson distributions).

The Percentile Method

The simplest way to form a Bootstrap CI is the Percentile Method. It’s very intuitive:
1. Sort your 1,000 bootstrap estimates from smallest to largest.
2. If you want a 95% Confidence Interval, find the values that cut off the bottom 2.5% and the top 2.5%.
3. For 1,000 samples, the 25th smallest value and the 975th smallest value form your confidence interval boundaries.

Common Mistake to Avoid: When doing the bootstrap in Paper B (using R), students sometimes forget to set the sample size \( n \) in the `sample()` function to be the same as the original dataset. Always resample the same number of observations!

5. Comparison: Asymptotics vs. Bootstrap

How do you decide which one to use?
Use Asymptotic MLE Properties when:
- The sample size \( n \) is large.
- You know the underlying distribution (e.g., you are told the data is Exponential).
- You can easily calculate the second derivative of the log-likelihood.

Use the Bootstrap when:
- The sample size is small.
- The distribution is unknown or very complex.
- You are using a computer (Paper B) to perform the simulation.

Chapter Summary

Key Takeaway 1: For large \( n \), the MLE \( \hat{\theta} \) is approximately \( N(\theta, \frac{1}{I(\theta)}) \). This is the "Gold Standard" for large-sample inference.
Key Takeaway 2: Fisher Information \( I(\theta) \) measures the curvature of the log-likelihood; more curvature means more information and less variance.
Key Takeaway 3: The Bootstrap is a computational tool that uses resampling with replacement to estimate the distribution of any statistic.
Key Takeaway 4: We can estimate bias, standard error, and confidence intervals (using the percentile method) directly from bootstrap samples without needing complex calculus.