Posterior distribution for a parameter in simple cases

Welcome to the World of Bayesian Updating!

Hello there! If you’ve ever changed your mind about something after seeing new evidence, you’re already thinking like a Bayesian. In this chapter of CS1, we are moving away from "Frequentist" statistics (where parameters are fixed numbers) and into Bayesian Statistics, where we treat parameters as random variables with their own distributions.

Don't worry if this seems a bit "meta" at first. We are simply learning a formal way to say: "I had a guess (Prior), I saw some data (Likelihood), and now I have a better guess (Posterior)." Let’s dive in!

1. The Core Ingredients: Bayes' Theorem

To find the Posterior Distribution, we use a specific version of Bayes' Theorem. Think of it as a recipe where we mix our prior beliefs with new information.

The Fundamental Formula

The relationship is defined as:
\( \pi(\theta | x) = \frac{f(x|\theta) \pi(\theta)}{f(x)} \)

However, in most CS1 exams, we use the Proportionality version because it’s much simpler to calculate:
\( \pi(\theta | x) \propto f(x|\theta) \times \pi(\theta) \)

In plain English:
Posterior \(\propto\) Likelihood \(\times\) Prior

Key Terms:
1. Prior Distribution \(\pi(\theta)\): What we believe about the parameter \(\theta\) before looking at any data.
2. Likelihood Function \(f(x|\theta)\): The probability of seeing our data given a specific value of \(\theta\).
3. Posterior Distribution \(\pi(\theta|x)\): Our updated belief about \(\theta\) after seeing the data \(x\).
4. Proportionality symbol (\(\propto\)): This means "is proportional to." We can ignore any constants that don't involve \(\theta\).

Quick Review: The "Dropping Constants" Trick

When calculating the posterior, if a term does not contain \(\theta\), you can toss it out! For example, if you have \( 3\theta^2 e^{-5\theta} \), the "3" is a constant and can be ignored during the derivation. This makes the math much less intimidating.

2. The Step-by-Step Recipe for Finding a Posterior

If you are asked to find the posterior distribution in an exam, follow these steps every single time:

Step 1: Write down the Prior. Identify \(\pi(\theta)\) and remove any constants that don't have \(\theta\) in them.
Step 2: Write down the Likelihood. This is \(f(x|\theta)\). If you have a sample of \(n\) observations, it is the product of the individual densities: \( \prod f(x_i|\theta) \).
Step 3: Multiply them together. Combine the terms from Step 1 and Step 2.
Step 4: Simplify. Group all the \(\theta\) terms together and all the \(e^{\dots \theta}\) terms together.
Step 5: Recognize the Distribution. Look at your simplified expression. Does it look like the "kernel" (the part with the variables) of a famous distribution like a Gamma, Beta, or Normal? (Check your Formulae and Tables book!)

Analogy: Imagine you are guessing how many red candies are in a jar. Your "Prior" is your guess based on the label. The "Likelihood" is what you see when you pull out 5 candies. The "Posterior" is your new, better guess.

3. Conjugate Priors: The "Shortcut" Pairs

In CS1, you will often encounter Conjugate Priors. This is just a fancy way of saying: "If the Prior and the Posterior belong to the same family of distributions, the Prior is conjugate to the Likelihood."

Knowing these pairs is like having a "cheat sheet" in your head. Here are the three most important ones for your syllabus:

A. The Poisson-Gamma Pair

If your data follows a Poisson(\(\theta\)) distribution and your prior is Gamma(\(\alpha, \lambda\)), then your posterior will also be a Gamma distribution.

The Update Rule:
New \(\alpha_{post} = \alpha + \sum x_i\)
New \(\lambda_{post} = \lambda + n\)
(Where \(n\) is the number of observations)

B. The Binomial-Beta Pair

If your data is Binomial(\(n, \theta\)) and your prior is Beta(\(a, b\)), your posterior will be Beta.

The Update Rule:
New \(a_{post} = a + \text{number of successes}\)
New \(b_{post} = b + \text{number of failures}\)

C. The Normal-Normal Pair (Fixed Variance)

If your data is Normal(\(\theta, \sigma^2\)) and your prior is Normal(\(\mu, \tau^2\)), your posterior is also Normal.

Note: The math for this one is a bit more algebraic, but the "New Mean" is essentially a weighted average of the prior mean and the sample mean.

Summary Takeaway: Conjugate priors make life easy because you don't have to do heavy integration; you just "update" the parameters of the distribution.

4. Summarizing the Posterior

Once you have found the posterior distribution, the examiners will often ask you for a Point Estimate. This is a single number that represents your "best guess" for \(\theta\).

1. The Posterior Mean

This is the expected value of the posterior distribution \(E[\theta | x]\).
Example: If your posterior is Gamma(\(\alpha', \lambda'\)), the mean is \(\alpha' / \lambda'\).

2. The Posterior Mode

This is the most likely value of \(\theta\) (the peak of the distribution). You find this by maximizing the posterior density (setting the derivative to zero).
Memory Aid: "Mode" = "Most." It's the most popular value!

3. The Posterior Median

The value that splits the posterior distribution in half. This is less common in exams but is found by solving for \(M\) where \(P(\theta \le M | x) = 0.5\).

5. Common Pitfalls to Avoid

1. Mixing up \(n\) and \(\sum x_i\): In the Poisson-Gamma update, students often forget to multiply \(\lambda\) by \(n\). Remember: \(\lambda\) deals with the sample size, while \(\alpha\) deals with the total count.

2. Forgetting the Likelihood product: If you have 10 data points, your likelihood is \(f(x_1|\theta) \times f(x_2|\theta) \dots f(x_{10}|\theta)\). Don't just use one \(x\)!

3. Panicking over the Constant of Proportionality: You don't need to calculate the denominator \(f(x)\) unless specifically asked. Focus on the terms with \(\theta\).

Final Quick Review Box

- Prior: Belief before data.
- Likelihood: What the data tells us.
- Posterior: Updated belief (Prior \(\times\) Likelihood).
- Conjugate: Prior and Posterior are the same type of distribution.
- Goal: Identify the distribution of \(\theta\) and use its properties (Mean/Mode) to make estimates.

Keep practicing these derivations! Once you recognize the patterns of the Gamma and Beta kernels, you’ll find these "simple cases" are some of the most rewarding marks to pick up in the CS1 exam. You've got this!