Welcome to the World of PCA Interpretation!
Hello there! If you’ve ever looked at a massive dataset with dozens of variables and felt completely overwhelmed, you are in the right place. In the world of Unsupervised Learning, Principal Components Analysis (PCA) is like a superpower that allows you to summarize complex data into a few key "themes."
In this chapter, we aren’t just running calculations; we are learning how to interpret what those calculations tell us about our data. By the end of these notes, you’ll be able to explain what a "loading" is, why scaling matters, and how to read those intimidating-looking Biplots. Let’s dive in!
1. The "Recipe" of PCA: Understanding Loadings
Imagine you are trying to describe a person’s health. You have data on their weight, height, blood pressure, and heart rate. Instead of looking at each number separately, you might want to create a single "Fitness Score." This score is a Principal Component (PC).
How do we build that score? We use Loadings.
Key Term: Loadings (\(\phi\))
The loadings tell us how much each original variable contributes to a specific Principal Component. Think of them as the "ingredients" in a recipe. If a variable has a high loading (close to 1 or -1), it is a very important part of that PC. If it’s close to 0, that variable doesn't care much about that PC.
The Loading Vector:
For the first principal component (PC1), the loading vector is written as:
\( \phi_{1} = (\phi_{11}, \phi_{21}, ..., \phi_{p1}) \)
Important Constraint:
To keep things fair, the sum of the squares of the loadings for a single PC must equal 1:
\( \sum_{j=1}^{p} \phi_{j1}^2 = 1 \)
This ensures the algorithm doesn't just make the loadings infinitely large to "fake" a high variance.
Key Takeaway: Loadings tell us the direction and importance of each variable in creating a Principal Component.
2. The "New Coordinates": Understanding Scores
Once we have our "recipe" (loadings), we apply it to our actual data points. The result is a Score.
Key Term: Principal Component Scores (\(z\))
The score is the value of a specific observation on a specific Principal Component. If PC1 represents "Overall Size," an elephant would have a very high score, while a mouse would have a very low score.
The Formula:
For observation \(i\), the score for the first principal component is:
\( z_{i1} = \phi_{11}x_{i1} + \phi_{21}x_{i2} + ... + \phi_{p1}x_{ip} \)
Don’t worry if this seems tricky! Just remember: The loadings define the component, and the scores are the individual data points transformed into that new component's "language."
3. Why Scaling is Non-Negotiable
This is a favorite topic for Exam SRM questions! Before performing PCA, we almost always scale our variables so they have a mean of 0 and a standard deviation of 1.
Why?
PCA is obsessed with variance. If one variable is measured in "Yen" (thousands) and another in "Percentage" (0 to 1), the Yen variable will have a much larger variance simply because the numbers are bigger. PCA will think the Yen variable is the most important thing in the world, even if it's not!
Common Mistake to Avoid:
Do not skip scaling unless all your variables are already in the same units (e.g., all are centimeters). If they are different units, always scale!
Quick Review:
- Unscaled: Variables with large units dominate.
- Scaled: Every variable gets a "fair vote."
4. Visualizing PCA: The Biplot
A Biplot is a special graph that shows both the Scores and the Loadings at the same time. It’s like a two-for-one deal!
How to read it:
1. The Points (Dots): These represent the individual observations (the scores). Points close together are similar to each other.
2. The Arrows (Vectors): These represent the original variables (the loadings).
3. Direction: If two arrows point in the same direction, those variables are positively correlated. If they point in opposite directions, they are negatively correlated.
4. Length: Longer arrows indicate that the variable has a stronger influence on the PCs shown in the plot.
Did you know? The axes of a Biplot are usually PC1 (x-axis) and PC2 (y-axis). PC1 will always capture more information than PC2.
5. Proportion of Variance Explained (PVE)
Since the whole point of PCA is to reduce data while keeping as much information as possible, we need a way to measure how much info we kept. This is the PVE.
Total Variance: This is the sum of the variances of all individual variables (usually equal to the number of variables \(p\) if we scaled them).
PVE of the \(m\)-th PC:
\( PVE_m = \frac{Variance \, explained \, by \, PC_m}{Total \, Variance} \)
The Scree Plot:
This is a simple bar or line graph showing the PVE for each component. Usually, it looks like a slide: high on the left, dropping down as you go right.
- The "Elbow" Method: To decide how many PCs to keep, look for the point where the drop-off levels out (the "elbow"). This is a common heuristic for choosing the right number of components.
Mnemonic: Think of PVE as "Power Value Earned." You want to earn as much "power" (variance) as possible with as few components as possible.
6. Summary and Final Tips
Summary Key Points:
- PC1 is the direction in which the data varies the most.
- Loadings define the direction; Scores are the projected data points.
- Scaling is essential to ensure units don't bias the results.
- Scree Plots help us choose the number of components by looking for an "elbow."
- PCA is an unsupervised technique—there is no "y" variable or "correct" answer we are predicting.
Study Tip for the Exam: If you see a question about PCA interpretation, first check if the variables were scaled. Then, look at the Biplot arrows to see which variables "move together." If PC1 explains 70% of the variance and PC2 explains 20%, you’ve captured 90% of the story with just two components!
Keep practicing—you're doing great! PCA is one of those topics that "clicks" suddenly after looking at a few examples.