Unsupervised learning techniques (principal component analysis, K-means clustering) to reduce data dimensionality, identify latent substructure and detect anomalies

Welcome to Machine Learning: Unsupervised Learning!

Hello there! Welcome to one of the most modern and exciting parts of the CS2 curriculum. So far in your studies, you might have looked at models where we try to predict a specific outcome (like whether a policyholder will make a claim). That is "Supervised Learning."

In this chapter, we are looking at Unsupervised Learning. Think of this as "detective work." We have a pile of data, but no specific "labels" or "target answers." Our goal is to find hidden patterns, group similar items together, and simplify complex information. It’s a vital tool for modern actuaries dealing with "Big Data."

Don't worry if this seems a bit abstract at first. We will break it down into simple steps with plenty of analogies!

1. Understanding Unsupervised Learning

In Unsupervised Learning, we provide the computer with input data \( (X) \), but no output labels \( (Y) \). The machine explores the data to find latent substructures (hidden patterns).

Why do we use it?
1. Dimension Reduction: Simplifying data with 100 variables into just 3 or 4 important ones.
2. Clustering: Grouping customers or risks that behave similarly.
3. Anomaly Detection: Finding data points that "don't fit," which might indicate fraud or unusual risk.

2. Principal Component Analysis (PCA)

Imagine you are trying to describe a person. You could list their height, weight, arm length, leg length, and shoe size. That’s a lot of variables! However, most of these are related to "size." PCA helps us combine these related variables into one single "Principal Component" called Overall Size.

What is PCA?

PCA is a technique used to reduce dimensionality. it transforms a large set of correlated variables into a smaller set of uncorrelated variables called Principal Components (PCs).

How it works (Step-by-Step):

1. Standardize the data: Because PCA is sensitive to the scale of measurement (e.g., meters vs. centimeters), we usually scale the data first so every variable has a mean of 0 and a variance of 1.
2. Find the First PC (\( PC_1 \)): This is a linear combination of the original variables that captures the maximum possible variance in the data.
3. Find the Second PC (\( PC_2 \)): This is another linear combination, orthogonal (at a right angle) to the first, capturing the next highest amount of variance.
4. Repeat: We continue until we have as many PCs as original variables.

The Math Behind It

A Principal Component is expressed as:
\( PC_1 = \phi_{1,1}X_1 + \phi_{2,1}X_2 + ... + \phi_{p,1}X_p \)
Where:
- \( X \) are our original variables.
- \( \phi \) (phi) are the loadings. These tell us how much weight each original variable has in the component.

Quick Review: Key Features of PCA

• The first PC always explains the most variance.
• Each subsequent PC is uncorrelated with the previous ones.
• We usually only keep the first few PCs that explain, say, 80% or 90% of the total variation.

Common Mistake: Forgetting to scale the data! If one variable is measured in millions and another in decimals, PCA will wrongly think the "millions" variable is the only one that matters.

Key Takeaway: PCA simplifies data by squashing many variables into a few "super-variables" while keeping as much information (variance) as possible.

3. K-means Clustering

While PCA is about variables, Clustering is about observations (the rows in your data). K-means clustering seeks to partition the data into \( K \) distinct, non-overlapping groups.

The Analogy

Imagine you have a bag of mixed coins. You want to sort them into 3 piles (where \( K=3 \)). You start by picking three spots on the table. You put each coin in the pile it is closest to. Then, you move the center of each pile to the middle of the coins you just moved. You repeat this until the piles stop moving!

The Algorithm:

1. Choose K: Decide how many clusters you want (e.g., \( K=3 \)).
2. Initialize: Randomly assign a "centroid" (center point) for each of the \( K \) clusters.
3. Assign: Look at each data point and assign it to the cluster whose centroid is closest (usually using Euclidean distance).
4. Update: Calculate the new mean (center) of all points in each cluster. This becomes the new centroid.
5. Iterate: Repeat steps 3 and 4 until the assignments no longer change.

How to choose "K"?

We use the Elbow Method. We run the algorithm for different values of \( K \) (1, 2, 3, 4...) and plot the "total within-cluster variation." As \( K \) increases, this variation drops. We look for the "elbow" or the "bend" in the graph where adding more clusters doesn't help much anymore.

Did you know? K-means is an iterative algorithm. Because it starts with random centroids, you might get slightly different results if you run it twice. Actuaries often run it multiple times and pick the best result!

Key Takeaway: K-means finds "clumps" of similar data points by minimizing the distance between points and their cluster center.

4. Latent Substructure and Anomaly Detection

Now that we know the tools, how do we use them to find "latent substructures" or "anomalies"?

Identifying Latent Substructure

A "latent substructure" is just a fancy way of saying a hidden group.
Example: You have data on motor insurance policyholders. You run K-means and find two distinct clusters. One cluster has high mileage and city driving; the other has low mileage and rural driving. The "substructure" is the driving environment, which wasn't explicitly labeled in your data!

Detecting Anomalies

Anomalies are outliers. We can find them using our two tools:
1. Via Clustering: If a data point is very far away from any cluster center, it is an anomaly.
2. Via PCA: If we use PCA to reduce data to two dimensions and one point sits far away from the "cloud" of other points, it's an anomaly.

Actuarial Example: In fraud detection, a claim that falls into a very tiny, isolated cluster or has a very unusual PCA score might be flagged for manual investigation.

5. Summary and Comparison

Let's wrap up with a quick comparison table to keep things clear:

Final Tips for the Exam:

• PCA: Remember that PCs are uncorrelated. This is a common exam question!
• K-means: Remember it is an unsupervised method. You don't tell the model what the groups are; it finds them itself.
• Scaling: Always mention that data should be standardized/scaled before performing these techniques.

Great job! You've just covered the essentials of Unsupervised Learning for CS2. Keep practicing with past paper questions to see how these concepts are tested in a numerical context!