Welcome to Principal Component Analysis (PCA)!
Welcome, future actuaries! Today we are diving into one of the most powerful tools in the Unsupervised Learning toolkit: Principal Component Analysis, or PCA.
Don't let the name intimidate you. If you’ve ever tried to summarize a 500-page book into a 1-page executive summary, you’ve already done a "mental PCA." In the world of data, we often have dozens (or hundreds) of variables. PCA helps us find a few "super-variables" that capture the essence of the data without all the noise. Let’s break it down step-by-step!
1. What exactly is PCA?
PCA is a technique used for dimension reduction. In many datasets, variables are correlated (related to each other). For example, in a dataset of house prices, "Square Footage" and "Number of Rooms" usually move together. Instead of looking at both, PCA finds a way to combine them into a single score.
The Goal: To find a low-dimensional representation of a dataset that contains as much of the variation as possible.
Analogy: Imagine you are trying to take a 2D photograph of a 3D statue. You want to stand at an angle where you can see the most detail and "depth." PCA finds that perfect angle for your data.
Quick Review: PCA is unsupervised because there is no "target" or "response" variable (\(y\)). We are only looking at the predictors (\(x_1, x_2, \dots, x_p\)).
2. The First Principal Component (\(PC_1\))
The First Principal Component is the specific linear combination of your variables that has the largest variance.
Mathematically, for a set of features \(x_1, x_2, \dots, x_p\), the first principal component score for the \(i^{th}\) observation is:
\( Z_{i1} = \phi_{11}x_{i1} + \phi_{21}x_{i2} + \dots + \phi_{p1}x_{ip} \)
Key Terms to Remember:
Loadings (\(\phi\)): These are the weights. For example, \(\phi_{11}\) is the loading of the first variable for the first principal component. They tell us how much "weight" each variable carries in the new component.
Scores (\(Z\)): These are the actual values of the transformed data points in the new "PCA space."
The Constraints:
To make the math work, we have a rule: the sum of the squares of the loadings must equal 1.
\( \sum_{j=1}^{p} \phi_{j1}^2 = 1 \)
Why? Without this rule, we could make the variance infinitely large just by picking huge weights! This constraint keeps things standardized.
Key Takeaway: The first principal component is the "direction" in the data along which the observations vary the most.
3. Calculating Subsequent Components
After we find the first principal component (\(PC_1\)), we can find the second (\(PC_2\)), the third, and so on.
The Rule of Orthogonality: The second principal component (\(PC_2\)) must be uncorrelated with \(PC_1\). In geometric terms, this means the direction of \(PC_2\) is perpendicular (orthogonal) to \(PC_1\).
Did you know? You can only have a maximum of \(min(n-1, p)\) principal components, where \(n\) is the number of observations and \(p\) is the number of variables.
Step-by-Step Calculation Logic:
1. Find the direction (\(PC_1\)) that captures the most variance.
2. Find the next direction (\(PC_2\)) that captures the most remaining variance, provided it is perpendicular to the first.
3. Repeat until you have accounted for all variance or reached the limit of components.
4. The Importance of Scaling
Don't skip this! This is a very common exam topic. Before you calculate PCA, you should almost always scale your variables to have a mean of zero and a standard deviation of one.
Why? PCA is sensitive to the units of measurement. Imagine a dataset with "Annual Income" (in dollars) and "Age" (in years). Income will have much larger numbers and a much larger variance just because of the scale. Without scaling, PCA will think Income is the only thing that matters simply because the numbers are bigger.
Quick Tip: If the variables are measured in the same units (e.g., all are test scores from 0-100), scaling might not be necessary. But in most SRM problems, standardizing is the safe and correct move.
Common Mistake to Avoid: Forgetting to center the data (subtract the mean). PCA looks for variance around the mean.
5. Proportion of Variance Explained (PVE)
How many principal components should we keep? To answer this, we look at the Proportion of Variance Explained (PVE).
We want to know: "How much of the total information in the original \(p\) variables is captured by the first \(m\) principal components?"
The total variance in a dataset is the sum of the variance of each individual variable. The PVE for the \(m^{th}\) component is:
\( PVE_m = \frac{\text{Variance explained by } PC_m}{\text{Total Variance}} \)
The Scree Plot
We often visualize PVE using a Scree Plot. It’s a simple graph where the x-axis is the component number and the y-axis is the PVE.
The "Elbow" Method: Look for the point where the PVE drops off significantly and the curve flattens out (the "elbow"). This is often the ideal number of components to keep.
Key Takeaway: Cumulative PVE tells us how much "total info" we have. If the first 3 components explain 95% of the variance, we can probably ignore the other 20 variables!
6. Summary and Final Tips
PCA is like a high-powered flashlight. It shines a light on the most important patterns in a messy, high-dimensional room.
Memory Aid (The 3 S's of PCA):
1. Scale the data first!
2. Scores are the new data points.
3. Scree plots help you decide when to stop.
Quick Review Box:
- Is PCA supervised or unsupervised? Unsupervised.- What do loadings represent? The weights of the original variables in the component.
- What is the relationship between PC1 and PC2? They are uncorrelated (orthogonal).
- Why scale? To prevent variables with large units from dominating the analysis.
Don't worry if the math feels heavy at first! For Exam SRM, focus on the concepts: why we use PCA (dimension reduction), the importance of scaling, and how to interpret PVE. You've got this!