Prior distribution, posterior distribution and conjugate prior distribution

Welcome to Bayesian Statistics!

In your CS1 journey so far, you have mostly used Frequentist statistics. This is where we assume a parameter (like a mean \(\mu\)) is a fixed, unknown number. But today, we are entering the world of Bayesian Statistics. In this world, we treat parameters as random variables. This means we can have "opinions" about them before we even see any data! Don't worry if this feels like a bit of a brain-flip; we'll take it one step at a time.

By the end of these notes, you will understand how to combine your initial beliefs with new evidence to form a more accurate conclusion. This is the heart of actuarial judgement!

1. The Three Musketeers: Prior, Likelihood, and Posterior

To understand Bayesian statistics, you need to know three main players. Let’s use an analogy: The Weather Forecast.

A. The Prior Distribution \(\pi(\theta)\)

The Prior Distribution represents what you believe about a parameter \(\theta\) before you collect any data. It’s your "gut feeling" based on past experience.

Example: Before looking outside, you think there is a 70% chance it will rain today because it is April in London.

B. The Likelihood Function \(f(x|\theta)\)

The Likelihood is the probability of seeing the data \(x\) that you actually observed, given that a specific parameter \(\theta\) is true. It’s the "new evidence."

Example: You look out the window and see dark, heavy clouds. The "Likelihood" tells us how often we see dark clouds when it is actually about to rain.

C. The Posterior Distribution \(\pi(\theta|x)\)

The Posterior Distribution is the updated belief about \(\theta\) after taking the new data into account. It is the result of combining your Prior and your Likelihood.

Example: Combining your knowledge of London in April with the dark clouds you just saw, you are now 95% sure it will rain.

Quick Review:
Prior: Belief before data.
Likelihood: What the data tells us.
Posterior: Updated belief after data.

2. The Golden Formula: Bayes' Theorem

In CS1, we use a specific version of Bayes' Theorem to find the Posterior distribution. The formal version looks like this:

\(\pi(\theta|x) = \frac{f(x|\theta)\pi(\theta)}{\int f(x|\theta)\pi(\theta) d\theta}\)

Wait! Don't panic! The denominator (the part at the bottom) is just a constant to make sure the probability adds up to 1. In most exam questions, we use the Proportionality Shortcut:

Posterior \(\propto\) Likelihood \(\times\) Prior

The symbol \(\propto\) means "is proportional to." This is your best friend in the exam. If you can multiply the Likelihood and the Prior and recognize the resulting "shape," you don't need to do the hard integration at the bottom!

3. Conjugate Prior Distributions

Sometimes, the math just "clicks" perfectly. A Conjugate Prior is a special type of prior distribution that, when combined with a specific likelihood, results in a posterior distribution that is in the same probability family as the prior.

Why is this useful?
If you start with a Gamma distribution and end with a Gamma distribution, you don't need to do any complex calculus. You just need to figure out how the parameters (like \(\alpha\) and \(\beta\)) have changed!

Did you know? Using conjugate priors is like a shortcut in a video game. It allows you to skip the "boss fight" (hard integration) and go straight to the reward (the posterior parameters).

4. The "Big Three" Conjugate Pairs to Remember

The IFoA exams frequently test three specific pairs. If you memorize these "updates," you will save a lot of time.

Pair 1: Poisson Likelihood + Gamma Prior

If your data \(x_1, ..., x_n\) follows a Poisson(\(\lambda\)) distribution and your Prior for \(\lambda\) is Gamma(\(\alpha, \beta\)):
The Posterior for \(\lambda\) is Gamma(\(\alpha + \sum x_i, \beta + n\)).

Memory Aid:
Add the sum of observations to \(\alpha\).
Add the number of observations to \(\beta\).

Pair 2: Binomial Likelihood + Beta Prior

If your data follows a Binomial(\(n, \theta\)) distribution (where \(\theta\) is the probability of success) and your Prior for \(\theta\) is Beta(\(\alpha, \beta\)):
The Posterior for \(\theta\) is Beta(\(\alpha + \text{successes}, \beta + \text{failures}\)).

Quick Tip: If the data is Bernoulli, it’s just a Binomial where \(n=1\). The update rule is the same!

Pair 3: Normal Likelihood + Normal Prior

If the data is Normal(\(\theta, \sigma^2\)) (with \(\sigma^2\) known) and the Prior for \(\theta\) is Normal(\(\mu_0, \sigma_0^2\)):
The Posterior is also Normal. The formulas for the updated mean and variance are a bit longer, but they essentially "weight" the prior mean and the sample mean based on their precision.

5. Step-by-Step: How to find a Posterior

If you are faced with a question that isn't a standard conjugate pair, follow these steps:

1. Write down the Prior \(\pi(\theta)\): Ignore any constants that don't involve \(\theta\).
2. Write down the Likelihood \(f(x|\theta)\): This is the joint density of the data. If you have \(n\) independent observations, multiply the individual densities together.
3. Multiply them together: \(\pi(\theta|x) \propto f(x|\theta) \times \pi(\theta)\).
4. Simplify: Group all the terms with \(\theta\) together and move everything else to the side.
5. Identify the Distribution: Look at the resulting expression. Does it look like the kernel of a Gamma? A Beta? A Normal? Match it to the formula sheet (Tables) to find the new parameters.

6. Common Mistakes to Avoid

1. Forgetting \(n\): In the Poisson-Gamma update, don't forget to use \(\beta + n\). If you have 10 years of data, \(n=10\).
2. Mixing up \(\alpha\) and \(\beta\): Always double-check your formula sheet for the Gamma and Beta definitions. Different textbooks sometimes swap the notation!
3. Over-calculating: Students often waste time calculating the denominator (the normalizing constant). Unless the question specifically asks for the "marginal distribution of x," you can usually ignore it by using the \(\propto\) sign.

Key Takeaways Summary

• Bayesian statistics updates initial beliefs with new data.
• Posterior \(\propto\) Likelihood \(\times\) Prior.
• A Conjugate Prior results in a Posterior of the same distribution family, making calculations simple.
• Poisson updates Gamma; Binomial updates Beta; Normal updates Normal.
• Always focus on the terms involving the parameter \(\theta\) and treat others as constants.

Don't worry if this feels tricky at first! Bayesian statistics is a different way of thinking. Once you practice the "parameter update" rules for the Big Three pairs, you'll find these are some of the most predictable marks in the CS1 exam.