Welcome to the World of Machine Learning!
Welcome to one of the most exciting (and modern) parts of the CFA Level II curriculum! While "Machine Learning" (ML) might sound like something out of a sci-fi movie, it is actually a powerful extension of the statistical tools you’ve already studied. In this chapter, we will learn how computers can "learn" from data to make predictions and find patterns without us telling them exactly what to look for. Don't worry if you aren't a computer scientist—we are going to break this down into simple, manageable pieces.
1. What exactly is Machine Learning?
At its core, Machine Learning is a subfield of Artificial Intelligence where algorithms are trained to find patterns in large datasets. Instead of a human writing a specific rule (like "If P/E < 15, then Buy"), the computer looks at the data and figures out the rules itself.
Supervised vs. Unsupervised Learning
The curriculum divides ML into two main "schools of thought" based on whether the data has labels:
A. Supervised Learning: Think of this like a student learning with a teacher. The dataset contains both the Inputs (Features) and the Correct Answers (Target Variables). The goal is to learn a rule that maps inputs to the correct output.
Example: Using historical data of "Company Ratios" (Inputs) and "Stock Returns" (Target) to predict future returns.
B. Unsupervised Learning: This is like a student exploring a library alone. The data has no labels and no target variable. The computer just looks for structure or groupings within the data.
Example: Giving a computer data on 500 stocks and asking it to group them into "similar" clusters based on their behavior, without telling it what the groups should be.
C. Reinforcement Learning: This is a bit different. An "agent" learns by trial and error to maximize a reward. It’s like training a dog with treats: good actions get a treat (reward), bad actions don't.
Quick Takeaway
Supervised: We know the answer (Target) we are looking for.
Unsupervised: We are just looking for hidden patterns or shortcuts in the data.
2. Supervised Learning: Regression and Classification
Supervised learning is further split into two categories based on what we are trying to predict:
1. Regression: Used when the target variable is continuous (a number).
Example: Predicting the specific price of a house or the exact return of a bond.
2. Classification: Used when the target variable is categorical (a label).
Example: Predicting if a company will "Default" or "Not Default."
Important Algorithms to Know:
Penalized Regression (LASSO): In standard regression, having too many variables can cause problems. LASSO (Least Absolute Shrinkage and Selection Operator) adds a penalty for having too many features. It can actually shrink the coefficients of unimportant variables to zero, effectively performing "feature selection" for you!
Support Vector Machines (SVM): Think of this as drawing the widest possible "boundary line" (a hyperplane) between two groups (e.g., "Success" vs. "Failure"). It tries to maximize the margin between the groups.
Classification and Regression Trees (CART): These are basically flowcharts. "Is the Debt/Equity > 1? If yes, go left. If no, go right." They are great because they are easy for humans to understand.
Random Forests: This is an Ensemble Method. Instead of relying on one tree (which might be wrong), we create hundreds of trees and let them "vote" on the outcome. This is the "Wisdom of the Crowd" approach.
3. Unsupervised Learning: Finding the Hidden Structure
Since we don't have "answers" in unsupervised learning, we use it for two main tasks:
Dimension Reduction (PCA)
Sometimes we have 100 different variables, but many of them tell us the same thing. Principal Component Analysis (PCA) summarizes the information from many variables into a few "Principal Components."
Analogy: Instead of describing a person by their height, arm length, leg length, and torso length (4 variables), you might just summarize it as "Size" (1 variable).
Clustering (K-Means)
K-Means Clustering groups data points into "K" number of clusters. It starts with random points and keeps refining the groups until the items in each group are as similar as possible to each other.
Analogy: Sorting a giant pile of laundry into piles of shirts, pants, and socks without being told what a "shirt" is.
Key Takeaway
PCA makes data smaller (fewer columns).
Clustering makes data organized (groups of rows).
4. Neural Networks and Deep Learning
Neural Networks (NNs) are inspired by the human brain. They consist of an Input Layer, Hidden Layers, and an Output Layer. When a network has many hidden layers, we call it Deep Learning.
Nets are incredibly powerful for non-linear relationships (complex patterns), but they are often called "Black Boxes" because it’s hard for a human to see exactly why the computer made a specific decision.
5. The Machine Learning Workflow
Building a model isn't just pressing a button. It follows a specific path:
1. Data Collection: Gathering the raw info.
2. Data Cleaning: Dealing with missing values and outliers.
3. Feature Engineering: Creating new variables (like turning a date into "Day of the Week").
4. Training/Validation/Testing: This is where students often get confused! Let's break it down.
The Three Datasets
To avoid "cheating," we split our data into three parts:
1. Training Set: The data the model actually "sees" and learns from.
2. Validation Set: Used to tune the model (like adjusting the "knobs" to find the best settings).
3. Test Set: The "Final Exam." The model only sees this once at the very end to see how it performs in the real world.
6. Overfitting and the Bias-Variance Trade-off
This is perhaps the most important concept for the exam!
Overfitting: This happens when the model learns the data too well—including the random noise and errors. It performs great on the training data but fails miserably on new data.
Analogy: Memorizing the exact answers to a practice exam instead of learning the math concepts. You’ll get 100% on the practice but fail the real exam.
Underfitting: The model is too simple to see the pattern.
Analogy: Trying to describe a complex curve with a straight line.
The Trade-off
\( \text{Total Error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Error} \)
High Bias = Underfitting: The model is too "stubborn" or simple.
High Variance = Overfitting: The model is too "flexible" or complex.
Memory Aid:
Variance = Very complex (Overfitting).
Bias = Basic (Underfitting).
7. Evaluating Model Performance
How do we know if our model is any good? For classification, we use a Confusion Matrix.
Key Metrics:
- Precision: Out of all the times the model predicted "Default," how many actually defaulted? (Quality over Quantity).
- Recall: Out of all the actual defaults that happened, how many did the model catch? (Quantity/Coverage).
- Accuracy: Total correct predictions divided by total predictions.
- F1 Score: A balance (harmonic mean) between Precision and Recall.
Common Mistake: Don't assume Accuracy is always the best metric. If 99% of companies don't default, a model that simply says "No one will ever default" will be 99% accurate but totally useless for finding the 1% who do!
Summary Checklist for Success
Before moving on, make sure you can answer these:
1. Can I explain the difference between Supervised and Unsupervised learning?
2. Do I know that LASSO helps with feature selection by zeroing out coefficients?
3. Do I understand that Overfitting leads to high variance and poor out-of-sample performance?
4. Can I identify that PCA is used for reducing the number of variables (dimension reduction)?
5. Do I remember that the Test Set is used only after the model is fully trained and tuned?
Don't worry if this feels like a lot of new vocabulary. Focus on the "Goal" of each method rather than the math behind it. The CFA Level II exam focuses heavily on your ability to choose the right tool for the right job!