Welcome to the World of Hierarchical Clustering!
In our journey through Unsupervised Learning, we previously looked at K-means clustering. While K-means is powerful, it has one major "annoyance": you have to tell the computer exactly how many clusters (K) you want before you even start! What if you don't know? Or what if you want to see how groups are nested within larger groups?
That is where Hierarchical Clustering shines. In this chapter, we will learn how to build a "tree" of data that allows us to pick the number of clusters after the analysis is done. It’s a favorite for actuaries because it provides a clear visual map of how data points relate to one another. Let’s dive in!
1. The Core Idea: Agglomerative Clustering
The most common version of hierarchical clustering is Agglomerative clustering. Don't let the big word scare you—it just means "bottom-up."
Think of it like a family reunion where everyone starts as a stranger. First, the two people most alike start a conversation. Then, the next two most similar people (or groups) join up. Eventually, everyone is part of one giant group.
How the Algorithm Works (Step-by-Step):
1. Start with \(n\) observations. Treat each individual observation as its own separate cluster (so you start with \(n\) clusters).
2. Calculate the dissimilarity (distance) between all pairs of clusters.
3. Find the two clusters that are the most similar (smallest distance) and fuse them together into a single new cluster. Now you have \(n-1\) clusters.
4. Repeat steps 2 and 3 until every observation is fused into a single, giant cluster containing everything.
Quick Tip: In hierarchical clustering, we aren't just grouping; we are recording the order and the distance at which clusters merge. This history is what makes the method so special.
2. The Dendrogram: Your Visual Map
The output of hierarchical clustering is a beautiful tree-like diagram called a Dendrogram. This is the most important tool for interpreting your results.
How to read a Dendrogram:
- The leaves at the bottom represent individual observations.
- As you move up the tree, leaves join into branches.
- The vertical height of the fusion (the horizontal line connecting two branches) represents how different those two clusters are. The higher the fusion, the more dissimilar the groups were.
- Crucial Rule: You cannot judge similarity based on how close leaves are to each other horizontally. You must look at the vertical height where they first share a common branch.
Choosing the Number of Clusters
Unlike K-means, we don't pick \(K\) at the start. Instead, we look at the dendrogram and decide where to "cut" the tree with a horizontal line. The number of vertical lines your cut crosses tells you how many clusters you have created.
Analogy: Imagine the dendrogram is a literal tree. If you saw off the trunk at a certain height, the number of separate branches falling to the ground is your number of clusters!
Key Takeaway: One single dendrogram can represent any number of clusters from 1 to \(n\). You just choose the "cut" height that makes the most sense for your business problem.
3. Linkage: How do we measure distance between groups?
We know how to measure distance between two points (usually Euclidean distance), but how do we measure the distance between a single point and a group of points? Or between two groups? This is called Linkage.
The four most common linkage types in the SRM curriculum are:
1. Complete (Max): The distance between two clusters is the distance between the two farthest points. This tends to produce well-separated, compact clusters.
2. Single (Min): The distance is the distance between the two closest points. Warning: This can lead to "chaining," where clusters look like long, thin lines rather than neat circles.
3. Average: The distance is the average of all pairwise distances between points in cluster A and cluster B. This is very popular because it's a "middle ground."
4. Centroid: The distance is measured between the centroids (the geometric centers) of the two clusters. Note: This can cause a weird glitch called an "inversion" where a fusion happens lower than a previous fusion, making the dendrogram hard to read.
Quick Review Box:
- Complete & Average: Generally preferred; they create more balanced trees.
- Single: Can create "trailing" clusters.
- Centroid: Used in some fields but can result in messy "inversions."
4. Choice of Dissimilarity Measure
While Euclidean distance (straight-line distance) is the default, it's not the only way. Sometimes we use Correlation-based distance.
When to use what?
- Use Euclidean distance if you care about the magnitude of the values (e.g., total sales volume).
- Use Correlation-based distance if you care about the pattern or shape of the data (e.g., do two stocks go up and down at the same time, even if one is \$10 and the other is \$100?).
Did you know? Data scaling is incredibly important here! If one variable is "Income" (thousands of dollars) and another is "Age" (years), the "Income" variable will dominate the distance calculation just because its numbers are bigger. Usually, we standardize variables to have a mean of 0 and a standard deviation of 1 before clustering.
5. Comparing Hierarchical and K-means Clustering
Students often ask, "Which one is better?" The answer is: "It depends!"
K-means is better when:
- You have a very large dataset (it’s usually faster).
- You have a clear idea of how many clusters you need.
Hierarchical is better when:
- You want to see the underlying structure/nested relationships.
- You don't want to commit to a specific \(K\) value upfront.
- You want a visual representation (dendrogram) to show stakeholders.
Common Pitfall: Don't assume hierarchical clustering is "more accurate" just because it's more complex to look at. Both methods can be "wrong" if the data doesn't actually have natural groups!
6. Summary and Final Tips
Hierarchical clustering is a flexible, visual way to group data. Remember these high-level points for your exam:
- It's Agglomerative (bottom-up).
- The Dendrogram is the primary tool for visualization.
- Height on the dendrogram represents dissimilarity.
- Linkage (Complete, Average, Single, Centroid) determines how we measure distance between clusters.
- Standardizing data is usually necessary to ensure all variables contribute equally.
Don't worry if the linkage types feel a bit abstract! Just remember that "Complete" looks for the furthest neighbors, and "Single" looks for the nearest. Most of the time, "Complete" and "Average" are the favorites for producing clean, useful trees.
You've got this! Hierarchical clustering is just a way of organizing a messy room by putting similar things together, then putting those groups into bigger boxes, until everything is in one big container. Keep that image in mind, and you'll master this chapter in no time!