MLE with truncated or censored data

Exam FAM – Fundamentals of Actuarial Mathematics · SOA (Society of Actuaries) · 5 min read

Welcome to the World of Incomplete Data!

Hi there! In your journey through Exam FAM, you've likely mastered Maximum Likelihood Estimation (MLE) for perfect, complete datasets. But in the real world of insurance, data is rarely perfect. Sometimes we only know a loss was "at least" a certain amount, or we don't even see the loss if it's below a deductible.

In this chapter, we will learn how to adjust our MLE "recipe" to handle Censored and Truncated data. Don't worry if these terms sound a bit technical—we'll break them down using simple analogies so you can tackle these exam questions with confidence!

1. Understanding the Vocabulary: Censoring vs. Truncation

Before we look at the math, we need to understand what happened to our data. If you can identify the type of data, the formula is easy!

Censoring: "I know you're there, but I don't know exactly how big you are."

In insurance, Right-Censoring is the most common type. This usually happens because of a policy limit.

Example: A policy has a limit of \$10,000. A massive fire causes \$50,000 in damage. The insurance company only records this as a \$10,000 claim. We know the actual loss was at least \$10,000, but we don't know the exact final number.

Truncation: "If you aren't big enough, I don't even know you exist."

In insurance, Left-Truncation is common because of deductibles.

Example: A policy has a \$500 deductible. If a policyholder has a \$300 loss, they won't file a claim. The insurance company never hears about it. Our data only includes people who "made it past" the \$500 threshold. We are missing the entire bottom part of the distribution.

Quick Review Box:

Censored: You have a data point, but its value is limited/capped.
Truncated: You are missing data points entirely because they fell below (or above) a certain value.

2. The "Building Blocks" of the Likelihood Function

To find the MLE, we need to build a Likelihood Function, $L(\theta)$. Think of this as a "probability product" where each piece of data contributes something to the total.

Case A: The Exact Observation

If we know the exact value $x$, the contribution is the probability density function (pdf):
Contribution = $f(x)$

Case B: Right-Censored at $u$

If we only know the value is at least $u$, we use the survival function. Why? Because the survival function $S(u)$ represents the probability of being greater than $u$.
Contribution = $S(u) = 1 - F(u)$

Case C: Left-Truncated at $d$

This is the trickiest one. Because we only see data given that it is greater than $d$, we must use a conditional probability. We divide the usual contribution by $S(d)$.
Contribution = $\frac{\text{f(x) or S(u)}}{S(d)}$

3. Putting it Together: The General Likelihood Formula

If we have a mix of exact values ($x_i$) and censored values ($u_j$), all subject to a truncation point ($d$), the Likelihood Function looks like this:

$L(\theta) = \prod_{i=1}^{n} \frac{f(x_i)}{S(d)} \times \prod_{j=1}^{m} \frac{S(u_j)}{S(d)}$

Wait! Don't panic! If there is no truncation, $d = 0$, and since $S(0) = 1$, the denominator just disappears. Most exam problems only have one or two of these elements at a time.

Did you know?
The most common mistake students make is forgetting to divide by $S(d)$ when a deductible is present. Always ask yourself: "Is there a value below which I wouldn't even see these claims?" If yes, that's truncation!

4. Step-by-Step Guide to Solving MLE Problems

Follow these steps to avoid getting lost in the algebra:

Step 1: Identify the components.
List out your exact values ($x$), your censored values ($u$), and your truncation point ($d$).

Step 2: Write the Likelihood Function $L(\theta)$.
Plug your values into the $f(x)$ and $S(x)$ formulas for your specific distribution (like Exponential or Pareto).

Step 3: Take the Natural Log to get the Log-Likelihood $l(\theta)$.
It is much easier to work with sums than products. Remember: $\ln(a \times b) = \ln(a) + \ln(b)$ and $\ln(a/b) = \ln(a) - \ln(b)$.

Step 4: Take the derivative with respect to $\theta$.
Set the derivative equal to zero: $\frac{d}{d\theta} l(\theta) = 0$.

Step 5: Solve for $\theta$.
This is your Maximum Likelihood Estimate!

5. Real-World Analogy: The "Dating App" Model

Imagine you are studying the height of people on a dating app to find the average height ($\theta$).

Exact data: People who list their height as exactly 5'10". (Use $f(x)$)
Censored data: People who just check a box that says "Over 6 feet tall." You don't know if they are 6'1" or 7'0". (Use $S(6)$)
Truncated data: The app has a filter where people shorter than 5 feet aren't allowed to join. You don't even see them in your data set! (Divide everyone by $S(5)$)

6. Common Pitfalls to Avoid

1. Mixing up $f(x)$ and $F(x)$: Remember, exact points get the pdf ($f$), and censored points (ranges) get the survival function ($S = 1-F$).

2. Forgetting the "n": When taking the log of a product like $\prod S(d)$, it becomes $n \ln S(d)$. Don't leave out the $n$!

3. Exponential Shortcuts: For the Exponential distribution with parameter $\theta$ (mean), the MLE is often just the (Average Excess Loss over the deductible). Look for these patterns to save time!

Key Takeaways Summary

1. Exact values contribute $f(x)$ to the likelihood.
2. Right-censored values (like policy limits) contribute $S(u)$.
3. Left-truncated values (like deductibles) require you to divide by $S(d)$.
4. The Goal: Maximize the log of the total product to find the best parameter $\theta$.

You're doing great! MLE with incomplete data is one of the "meatier" topics in FAM, but once you recognize the patterns of $f(x)$, $S(u)$, and $S(d)$, it becomes a simple game of assembly. Keep practicing!

Quick check

Can you answer these now?

Open each question to check the key ideas from this chapter.

What is right-censoring in the context of actuarial data?

Right-censoring occurs when an observation is only known to be at least a certain value (usually a policy limit), but the actual value beyond that limit is unknown.

What is left-truncation and how does it differ from censoring?

Left-truncation occurs when values below a threshold (like a deductible) are completely omitted from the dataset; the insurer never knows these losses existed.

What is the likelihood contribution of an exact, non-censored observation $x$?

The contribution of an exact observation is the probability density function (pdf) evaluated at that point: $f(x)$.

What is the likelihood contribution of a right-censored observation at point $u$?

The contribution is the survival function evaluated at the censored point: $S(u) = 1 - F(u)$.

How do you adjust the likelihood contribution for a left-truncated observation at deductible $d$?

You divide the standard contribution (either $f(x)$ or $S(u)$) by the survival function at the truncation point: $S(d)$.

What is the general Likelihood Function formula for mixed censored and truncated data?

$L(\theta) = \prod_{i=1}^{n} \frac{f(x_i)}{S(d)} \times \prod_{j=1}^{m} \frac{S(u_j)}{S(d)}$

Why is the Log-Likelihood used instead of the standard Likelihood function?

Taking the natural log converts the product of probabilities into a sum, making it much easier to differentiate and solve for the parameter $\theta$.

In the log-likelihood of a truncated dataset with $N$ observations, how is the truncation term $S(d)$ handled?

It becomes a subtracted term multiplied by the total number of observations: $-N \cdot \ln[S(d)]$.

Ready to test yourself?

Turn these notes into exam-style practice. Get unlimited AI questions on this topic with instant marking and explanations.

Practice This Topic

More Exam FAM – Fundamentals of Actuarial Mathematics chapters

* The content provided by thinka is generated by AI and may not always be accurate or up-to-date. Please use it as a supplementary resource and verify with official materials.

Put These Notes into Practice

Reading the notes is a great start. Now practise with unlimited AI-generated questions and get instant feedback. 100,000+ students are already improving their grades.

Start Practising Now View Pricing

Done reading? Test yourself with AI practice questions

Practice This Topic Now