K-means and hierarchical clustering for data transformation

Welcome to Clustering for Data Transformation!

Hi there! Welcome to one of the most creative parts of Exam PA. Usually, when we think of clustering, we think of it as the "end goal"—like grouping customers into segments. But in this chapter, we are looking at clustering as a tool to help our other models (like GLMs or Trees) work better. This is what we call Data Transformation. We are going to learn how to take raw data and turn it into "cluster labels" that act as powerful new features for our predictive models.

Don't worry if unsupervised learning feels a bit abstract right now. We’ll break it down step-by-step using simple analogies. Let’s get started!

1. The Basics: What is Clustering?

Before we use clusters to transform data, we need to understand what they are. In Unsupervised Learning, we don't have a "target" variable (the y). We only have our features (the x's). Clustering is the process of finding natural groupings in the data so that items in the same group are more similar to each other than to those in other groups.

Quick Review: Think of a messy room. Clustering is the act of putting all the "socks" in one pile and "shirts" in another, even if nobody told you which is which. You just noticed that all the socks look similar to each other!

2. K-means Clustering

K-means is the most popular clustering algorithm. It tries to partition the data into K distinct, non-overlapping groups.

How K-means Works (Step-by-Step)

1. Pick K: You decide how many clusters you want (let's say K=3).
2. Initialize: The computer randomly picks 3 points to be the "starting centers" (called centroids).
3. Assign: Every data point looks at those 3 centers and joins the one it is closest to.
4. Update: The centers move to the middle of their new groups.
5. Repeat: Steps 3 and 4 keep happening until the centers stop moving.

The Importance of Scaling

Critical Point: K-means uses Euclidean Distance to decide who belongs where. The formula looks like this:
\( d(x, y) = \sqrt{\sum_{i=1}^{n} (x_i - y_i)^2} \)
Because it uses distance, you must scale (standardize) your data first! If one variable is "Annual Income" (thousands of dollars) and another is "Age" (0-100), the income variable will completely dominate the distance calculation. Scaling puts them on a level playing field.

Common Mistake: Forgetting to scale before K-means is a very common error on the PA exam. Always check if your variables are on different scales!

Choosing the right "K"

How do we know if K should be 2, 3, or 10? We use the Elbow Method. We plot the "Total Within-Cluster Sum of Squares" (a measure of how spread out the clusters are) against the number of clusters. We look for the "elbow"—the point where adding more clusters doesn't help much anymore.

Key Takeaway: K-means is fast and simple, but you have to choose K in advance, and it works best when clusters are spherical (roundish) blobs.

3. Hierarchical Clustering

Unlike K-means, hierarchical clustering doesn't require you to pick "K" at the start. Instead, it builds a tree-like structure.

Agglomerative (Bottom-Up) Clustering

Think of this like a family tree in reverse. Every data point starts as its own tiny cluster. Then, the two closest points merge into a pair. Then, the next closest merge, and so on, until everyone is in one giant group.

The Dendrogram

The result is a Dendrogram—a beautiful diagram that looks like an upside-down tree. To choose your number of clusters, you simply "cut" the tree horizontally. Where you cut determines how many clusters you get!

Linkage: How do we measure distance between groups?

When merging two groups of points, we need a rule for distance. This is called Linkage:
• Complete Linkage: Uses the distance between the farthest points in the clusters.
• Single Linkage: Uses the distance between the closest points (can lead to "stringy" clusters).
• Average Linkage: Uses the average distance between all pairs.
• Ward’s Method: Minimizes the variance within clusters (very popular in actuarial work because it creates groups of similar sizes).

Key Takeaway: Hierarchical clustering is great when you want to see the relationship between groups, but it can be slow on very large datasets.

4. Using Clusters for Data Transformation

This is the most important part for Exam PA! Once we have our clusters, how do we use them to build a better model?

The "Cluster Membership" Feature

Once the algorithm is done, every row in your data gets a label (e.g., Cluster 1, Cluster 2, or Cluster 3). You can add this label as a new categorical variable in your dataset.

Why does this help?
1. Capturing Non-linearities: A simple GLM might struggle with complex patterns. By grouping similar observations together, the cluster label can capture those "pockets" of behavior that a linear term might miss.
2. Identifying Interactions: Clusters often represent a combination of features (e.g., "Young people with high debt"). Instead of manually creating an interaction term between Age and Debt, the cluster label does it for you!
3. Dimensionality Reduction: Sometimes, instead of using 10 confusing variables, using 1 cluster label that summarizes those 10 variables can make the model simpler and easier to explain to stakeholders.

Real-World Example: If you are predicting auto insurance claims, a cluster might represent "Urban high-mileage drivers." Even if your model doesn't have a specific interaction for "Location * Mileage," the cluster label acts as a shortcut to tell the model: "Hey, this group is high risk!"

5. Summary and Best Practices

Quick Review Box:
• K-means: Fast, needs K specified, needs scaling, creates "blob" groups.
• Hierarchical: Visual (Dendrogram), no K needed at start, choice of linkage matters.
• Transformation: We use cluster assignments as a new categorical feature in a supervised model (like a GLM or GBM).
• Scaling: Always standardize your numeric data before clustering!

Did you know? Clustering is often called "finding the hidden structure." In Exam PA, your job is to find that structure and use it to give your predictive model a "hint" about which observations are similar.

Final Encouragement: If you find the different linkage types or the math behind Euclidean distance confusing, don't sweat the small stuff. Focus on why we cluster: to simplify complex data and create new features that help our models see patterns more clearly. You’ve got this!