Empirical Bayesian estimation: nonparametric and semiparametric cases

Welcome to Empirical Bayesian Estimation!

In our previous studies of credibility, we often assumed we knew the underlying distribution of our data (like Poisson or Normal). But in the real world, we rarely have that luxury. This is where Empirical Bayesian Estimation steps in. It’s like being a detective—instead of assuming what the criminal looks like, we look at the evidence (the data) to figure out the patterns.

In this chapter, we focus on nonparametric and semiparametric cases. Don't let those big words scare you! "Nonparametric" simply means we don't assume a specific distribution type, and "semiparametric" means we know a little bit about the structure but not everything. We are going to learn how to estimate the "Big Three" parameters: \(\mu\) (the overall mean), \(v\) (the process variance), and \(a\) (the variance of the hypothetical means).

1. The Big Idea: Why Empirical Bayes?

The goal of credibility is to find the formula: \(Z \bar{X} + (1-Z)\mu\). To calculate the credibility factor \(Z\), we need two things: the Expected Value of the Process Variance (v) and the Variance of the Hypothetical Means (a).

Nonparametric estimation is used when we have data for several different policyholders (or "risks") over several years, but we have no idea what the underlying probability distribution is. We use the data itself to estimate \(v\) and \(a\).

Analogy: Imagine you are a coach of a track team. You don't know the exact "talent distribution" of all humans (the parametric model), but you have the race times of your 10 athletes over the last 5 races. You use those specific times to estimate how much an athlete's performance varies race-to-race (\(v\)) and how much the athletes' talents differ from each other (\(a\)).

2. The Nonparametric Case (Bühlmann Model)

In this scenario, we assume every risk (policyholder) has the same amount of exposure (e.g., each person is one "unit"). We have \(k\) risks and each risk has \(n\) years of data.

Step-by-Step Parameter Estimation

Step 1: Calculate the Average for each Risk
First, find the mean for each individual policyholder (\(i\)):
\(\bar{X}_i = \frac{1}{n} \sum_{j=1}^{n} X_{ij}\)

Step 2: Calculate the Grand Mean (\(\hat{\mu}\))
This is just the average of all the individual averages:
\(\hat{\mu} = \bar{X} = \frac{1}{k} \sum_{i=1}^{k} \bar{X}_i\)

Step 3: Estimate the Process Variance (\(\hat{v}\))
This measures how much an individual's claims vary from their own average. We average the sample variances of all risks:
\(\hat{v} = \frac{1}{k(n-1)} \sum_{i=1}^{k} \sum_{j=1}^{n} (X_{ij} - \bar{X}_i)^2\)

Step 4: Estimate the Variance of Hypothetical Means (\(\hat{a}\))
This measures how much the "true means" of different risks differ from each other. It’s a bit trickier because the raw variance between group means also includes some "noise" from the process variance:
\(\hat{a} = \frac{1}{k-1} \sum_{i=1}^{k} (\bar{X}_i - \bar{X})^2 - \frac{\hat{v}}{n}\)

Quick Review: Why do we subtract \(\frac{\hat{v}}{n}\)? Because the variation we see between the policyholders' sample means is "inflated" by the fact that each sample mean is just an estimate. We subtract the noise to find the "true" variance between the individuals.

Summary Takeaway: If \(\hat{a}\) is large, it means policyholders are very different from each other, so the individual data is very useful (high \(Z\)). If \(\hat{a}\) is small, everyone is pretty much the same, so we rely more on the group average (low \(Z\)).

3. The Semiparametric Case (Bühlmann-Straub Model)

What if the risks are different sizes? For example, one "risk" might be a small business with 5 employees, and another is a corporation with 500. This is the Bühlmann-Straub model. We now have weights (exposures) denoted as \(m_{ij}\).

Estimation with Different Exposures

The Mean (\(\hat{\mu}\)):
We use a weighted average of all claims:
\(\hat{\mu} = \frac{1}{m} \sum_{i=1}^{k} \sum_{j=1}^{n} m_{ij} X_{ij}\), where \(m\) is the total exposure.

The Process Variance (\(\hat{v}\)):
\(\hat{v} = \frac{1}{\sum_{i=1}^{k} (n_i - 1)} \sum_{i=1}^{k} \sum_{j=1}^{n} m_{ij} (X_{ij} - \bar{X}_i)^2\)

The Variance of Hypothetical Means (\(\hat{a}\)):
This formula is the "Final Boss" of the chapter. It looks intimidating, but it follows the same logic as the nonparametric case:
\(\hat{a} = \frac{\sum_{i=1}^k m_i (\bar{X}_i - \bar{X})^2 - (k-1)\hat{v}}{m - \frac{1}{m} \sum_{i=1}^k m_i^2}\)
Where \(m_i\) is the total exposure for risk \(i\).

Did you know? In some problems, the math might result in a negative value for \(\hat{a}\). Since variance cannot be negative, if this happens in an exam, we set \(\hat{a} = 0\). This means the individual data is no more useful than the group average, and \(Z = 0\).

4. Common Pitfalls and Memory Aids

Don't mix up your denominators!
- When calculating \(\hat{v}\) in the nonparametric case, the denominator is \(k(n-1)\). Think: "Number of groups times the degrees of freedom in each group."
- When calculating the sample variance for \(\hat{a}\), the denominator is \(k-1\). Think: "Comparing the \(k\) groups to each other."

The "Zero Variance" Trap:
If \(\hat{a}\) is calculated as negative, always set \(\hat{a} = 0\). If you forget this, your \(Z\) will be negative, which makes no sense in the context of credibility! You can't have "negative" belief in your data.

Memory Trick:
For the Bühlmann-Straub \(\hat{a}\) formula, notice the denominator: \(m - \frac{\sum m_i^2}{m}\). This looks a lot like the formula for variance \((E[X^2] - (E[X])^2)\). This is because \(\hat{a}\) is trying to isolate the variance between the groups.

Summary of Key Takeaways

1. Nonparametric: Used when all risks have equal exposure and we don't know the distribution. We use averages and sample variances to find \(v\) and \(a\).
2. Semiparametric: Used when risks have different exposures (Bühlmann-Straub). The formulas get heavier with weights (\(m_{ij}\)), but the logic remains: Variance between groups minus noise.
3. Unbiased Estimators: All these estimators (\(\hat{\mu}, \hat{v}, \hat{a}\)) are designed to be "unbiased," meaning on average, they hit the true value.
4. The "Negative a" Rule: If your calculation for \(\hat{a}\) is \(\leq 0\), then \(\hat{a} = 0\) and \(Z = 0\).

Don't worry if these formulas seem long! Most exam questions provide the sums for you (like \(\sum \sum (X_{ij} - \bar{X}_i)^2\)). Your job is to know which formula to plug them into. Practice a few problems, and you'll start to see the pattern!