Understand and apply a Markov chain

Welcome to the World of Markov Chains!

Hello there! Today, we are diving into one of the most elegant and useful tools in an actuary's toolkit: Markov Chains. Whether you are calculating the probability of a driver moving between "No Claims Discount" levels or predicting if a policyholder will remain healthy, Markov Chains are your go-to model.

Don't worry if stochastic processes sound intimidating. At its heart, a Markov Chain is just a way of looking at how things change over time when the future depends only on where we are right now. Let’s break it down step-by-step!

1. What is a Markov Chain?

A Markov Chain is a stochastic process (a collection of random variables indexed by time) that moves between different "states" in a discrete state space.

The "Golden Rule": The Markov Property

The most important thing to remember is the Markov Property. It states that the future behavior of the process depends only on its present state, and not on the sequence of events that preceded it. In other words, the process has no memory.

\( P(X_{n+1} = j | X_n = i, X_{n-1} = i_{n-1}, ..., X_0 = i_0) = P(X_{n+1} = j | X_n = i) \)

Analogy: The Frog on Lily Pads
Imagine a frog jumping between lily pads in a pond. If the frog is on Lily Pad A, the probability of its next jump to Lily Pad B depends only on the fact that it is currently on A. It doesn't matter if the frog arrived at A from Lily Pad C or Lily Pad D. The history doesn't change the next jump!

Quick Review:
- State Space (S): The set of all possible "locations" or "statuses" (e.g., {Healthy, Sick, Dead}).
- Discrete Time: We look at the process at specific steps (Time 0, 1, 2...).

2. Transition Probabilities and the TPM

How do we move from one state to another? We use transition probabilities.

One-Step Transition Probabilities

The probability of moving from state \( i \) to state \( j \) in exactly one time step is written as:
\( p_{ij} = P(X_{n+1} = j | X_n = i) \)

The Transition Probability Matrix (TPM)

When we have multiple states, we organize these probabilities into a square matrix, usually called \( P \). Each row represents where you are starting, and each column represents where you are going.

Important Rules for the TPM:
1. Every entry must be between 0 and 1 (they are probabilities!).
2. The sum of every row must equal 1. This is because the process must go somewhere (even if it stays in the same state).

Did you know?
In the IFoA exams, a common mistake is to sum the columns instead of the rows. Always double-check: Rows = "From", Columns = "To", Row Total = 1.

3. Time-Homogeneity

In CS2, we often deal with Time-Homogeneous Markov Chains. This just means the rules don't change over time. The probability of moving from State A to State B is the same today as it will be next week.

Example: A fair coin toss is time-homogeneous. The probability of getting heads is always 0.5.
Non-example: Human mortality is not time-homogeneous. The probability of "moving" from the state "Alive" to "Dead" in the next year increases as you get older.

4. Moving Further: Multi-Step Transitions

What if we want to know the probability of being in state \( j \) after two steps, or \( n \) steps?

The Chapman-Kolmogorov Equations

These equations tell us that to get from \( i \) to \( j \) in \( m+n \) steps, we must pass through some intermediate state \( k \) at step \( m \).

\( p_{ij}^{(m+n)} = \sum_{k \in S} p_{ik}^{(m)} p_{kj}^{(n)} \)

The "Magic" Trick:
To find the \( n \)-step transition matrix, you simply raise the one-step matrix to the power of \( n \)!
\( P^{(n)} = P^n \)

So, if you want the 2-step probabilities, calculate \( P \times P \). If you want 3 steps, calculate \( P \times P \times P \).

5. Classifying States

Not all states are created equal. We classify them to understand the "long-term" behavior of the chain.

• Irreducible: A chain is irreducible if you can eventually get from any state to any other state. There are no "traps" or isolated islands.
• Periodicity: A state has period \( d \) if the process can only return to it in multiples of \( d \) steps. If \( d=1 \), the state is aperiodic. (Think of a clock: it only hits "12" every 12 hours).
• Recurrent vs. Transient:
  - Recurrent: If you start here, you are certain (probability 1) to return eventually.
  - Transient: There is a chance you might leave and never come back.

Memory Aid:
Think of an Absorbing State (like "Dead" in a life insurance model) as a black hole. Once you enter, you can never leave. This is a special type of recurrent state.

6. Stationary Distributions (\( \pi \))

If we let the Markov Chain run for a very long time, does it settle down into a steady pattern? This "steady state" is called the Stationary Distribution, denoted by the vector \( \pi \).

To find \( \pi \), we solve these equations:
1. \( \pi P = \pi \) (The distribution stays the same after another step)
2. \( \sum \pi_i = 1 \) (The total probability must be 1)

Step-by-Step: Solving for \( \pi \)
1. Set up the matrix equation \( \pi P = \pi \). This gives you a system of linear equations.
2. One of these equations will be redundant (useless), so throw one away.
3. Replace the discarded equation with the "normalizing condition": \( \pi_1 + \pi_2 + ... + \pi_n = 1 \).
4. Solve for the unknowns.

Key Takeaway:
If a chain is irreducible and aperiodic, the distribution will eventually converge to this stationary distribution \( \pi \) regardless of where you started!

7. Common Pitfalls to Avoid

1. Forgetting the "No Memory" Rule: If an exam question mentions that the probability changes based on how long you've been in a state, it might not be a simple Markov Chain! (You might need to create more states to make it Markovian).
2. Matrix Multiplication Order: Remember, it is usually \( \text{Row Vector} \times \text{Matrix} \). Order matters in matrix algebra!
3. Skipping the Sum-to-One Check: Always make sure your final \( \pi \) values add up to exactly 1.

Don't worry if this seems tricky at first! The math often looks scarier than it is. Once you get used to "jumping" between states using the TPM, you'll see that Markov Chains are just a logical way to map out the future.

Summary Review

The Markov Property: Only the present matters.
TPM: Rows sum to 1. \( P_{ij} \) is "from \( i \) to \( j \)".
n-step: Use matrix powers (\( P^n \)).
Stationary Distribution: Solve \( \pi P = \pi \) and \( \sum \pi = 1 \).
States: Can be irreducible, periodic, recurrent, or transient.