Welcome to the World of GLMs!

Hello there! If you’ve been studying linear regression, you know it’s a powerful tool. But what happens when your data doesn't follow a perfect "bell curve" (Normal distribution)? What if you're modeling the number of car accidents (which can't be negative) or whether a life insurance policy will lapse (yes/no)?

That is where Generalized Linear Models (GLMs) come in. In this chapter, we are going to look at the "skeleton" of GLMs: The Exponential Family and Link Functions. Think of the exponential family as a VIP club for probability distributions, and the link function as the secret handshake that lets them work with linear models. Let's dive in!

1. What is the Exponential Family?

The exponential family is a group of probability distributions that share the same mathematical structure. By putting them in this specific format, we can use one single framework to solve many different types of problems.

A distribution belongs to the exponential family if its probability density (or mass) function can be written like this:

\( f(y; \theta, \phi) = \exp \left( \frac{y\theta - b(\theta)}{a(\phi)} + c(y, \phi) \right) \)

Don't let the notation scare you! Let’s break down the "players" in this equation:

  • \( y \): This is your observed data (the response variable).
  • \( \theta \) (Theta): This is called the canonical parameter (or natural parameter). It is related to the mean of the distribution.
  • \( \phi \) (Phi): This is the dispersion parameter. It relates to how spread out the data is (the scale).
  • \( a(\phi) \), \( b(\theta) \), and \( c(y, \phi) \): these are just functions that change depending on which distribution you are using (Normal, Poisson, etc.).

Why is this useful?

Analogy: Imagine a shipping company that only handles boxes of a specific size. If you want to ship a car, a piano, or a bag of marbles, you have to pack them into that standard box. Once they are in the box, the shipping company doesn't care what's inside—they treat every box the same way. The Exponential Family formula is that "standard box" for statistics!

Quick Review: Key Parts

\( b(\theta) \) is the most important function to remember because it determines the mean and the variance! We call it the cumulant function.


2. Finding the Mean and Variance

One of the coolest things about the exponential family is that once you find \( b(\theta) \), you can find the mean and variance using simple derivatives. You don't have to do any long integrations!

The Mean:
The expected value of \( Y \) (the mean, \( \mu \)) is simply the first derivative of \( b(\theta) \):
\( E[Y] = \mu = b'(\theta) \)

The Variance:
The variance of \( Y \) involves the second derivative of \( b(\theta) \) and the dispersion function:
\( Var(Y) = b''(\theta) a(\phi) \)

Did you know? The term \( b''(\theta) \) is often called the variance function, denoted as \( V(\mu) \). It describes how the variance changes as the mean changes. In a Normal distribution, the variance is constant. But in a Poisson distribution, the variance gets bigger as the mean gets bigger!


3. Common Members of the Family

You need to recognize which famous distributions live in this family. For Exam SRM, these are the most common:

  • Normal (Gaussian): Used for standard continuous data.
  • Poisson: Used for counts (e.g., how many claims occur in a year).
  • Binomial: Used for binary outcomes (e.g., did the person pass the exam? Yes or No).
  • Gamma and Inverse Gaussian: Used for continuous, positive data that is "skewed" (e.g., the dollar amount of an insurance claim).

Common Mistake: Students often forget that the standard Linear Regression is actually just a special case of a GLM where the distribution is Normal!


In a standard linear model, we say \( Y = \beta_0 + \beta_1 X \). But what if \( Y \) can only be positive (like claim amounts), but our formula \( \beta_0 + \beta_1 X \) produces a negative number?

This is where the Link Function comes in. It "links" the mean of our data (\( \mu \)) to the linear predictor (\( \eta \)).

The Formula:
\( g(\mu) = \eta = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + ... \)

Analogy: The Link Function is like a language translator. The linear model speaks "Real Numbers" (anything from negative infinity to positive infinity). The data speaks "Probabilities" (only 0 to 1) or "Counts" (only positive integers). The link function translates the model's output so it makes sense for the data.

Important Link Functions to Know:

1. Identity Link: \( g(\mu) = \mu \). (Used for Normal distributions). It doesn't change anything.
2. Log Link: \( g(\mu) = \ln(\mu) \). (Used for Poisson). This ensures that our predicted mean is always positive because \( \mu = e^\eta \).
3. Logit Link: \( g(\mu) = \ln(\frac{\mu}{1-\mu}) \). (Used for Binomial). This ensures our predicted probability is always between 0 and 1.


Every distribution in the exponential family has one "natural" link function that is mathematically the easiest to use. This is called the Canonical Link Function.

It occurs when we set the link function equal to the canonical parameter we saw earlier:
\( g(\mu) = \theta \)

Quick Reference Table:

Normal: Identity Link
Poisson: Log Link
Binomial: Logit Link
Gamma: Inverse Link (\( 1/\mu \))

Don't worry if this seems tricky! Just remember: "Canonical" usually means "the default" or "the mathematically cleanest" option for that distribution.


6. Summary and Key Takeaways

You've made it through the core of GLM theory! Here is what you should take away for the exam:

  • The Exponential Family is a general form that includes Normal, Poisson, Binomial, and Gamma distributions.
  • \( b(\theta) \) is your best friend. Its first derivative is the mean, and its second derivative helps you find the variance.
  • The Link Function \( g(\mu) \) connects the mean to the linear predictors (\( X\beta \)).
  • The Canonical Link is the specific function where \( g(\mu) = \theta \).
  • Memory Trick: To remember the Logit link for Binomial, think of "Probability" -> "Odds" -> "Log-Odds." Log-odds is the Logit!

Keep going! Understanding how these pieces fit together is the hardest part. Once you have the "skeleton" down, the rest of the GLM chapters will feel much more intuitive. You've got this!