Big Data Projects

CFA Level II · CFA · 6 min read

Welcome to the World of Big Data Projects!

Hello there! If you’ve made it to CFA Level II, you already know that data is the lifeblood of modern finance. In this chapter, we move beyond simple spreadsheets and dive into the structured workflow of Big Data Projects. Think of this as the "instruction manual" for how investment firms take raw, messy data and turn it into actionable investment signals. Don’t worry if you aren’t a computer scientist—we are going to break this down step-by-step using logic and everyday analogies.

1. The Big Data Project Lifecycle

A Big Data project isn't just about "crunching numbers." It follows a specific, logical sequence. If you miss a step, the whole analysis might fall apart. Think of it like baking a complex cake: if you don’t prep the ingredients correctly, it doesn't matter how good your oven is!

The standard workflow involves these five major steps:
1. Conceptualizing the Project: Defining the problem.
2. Data Collection: Gathering the raw materials.
3. Data Preparation (Wrangling): Cleaning and organizing.
4. Model Building: The actual "Machine Learning" part.
5. Model Evaluation: Checking if the results are actually useful.

Quick Tip: Data Preparation is often cited as taking up to 80% of a data scientist's time. In the CFA exam, pay close attention to the cleaning steps!

2. Step 1 & 2: Conceptualizing and Data Collection

Before you touch any data, you must ask: "What am I trying to solve?" Are you trying to predict stock prices? Detecting fraudulent trades? Once the goal is set, you collect the data.

Types of Data Sources

In Big Data projects, we deal with two main types of data:
- Structured Data: Think of a standard Excel sheet. It has rows and columns (e.g., historical stock prices, P/E ratios).
- Unstructured Data: This is "messy" data like news articles, social media posts, or corporate earnings call transcripts. This usually requires Natural Language Processing (NLP) to understand.

Key Takeaway: The foundation of any project is a clear objective and a diverse set of reliable data sources.

3. Step 3: Data Preparation (The "Cleaning" Phase)

Raw data is almost always "dirty." It has errors, missing values, and weird outliers. We use Data Wrangling (or Preprocessing) to fix this.

A. Handling Missing Data

If a data point is missing, you have two main choices:
- Exclusion: Just delete the row. (Careful! If you delete too many, you lose valuable info).
- Imputation: Fill in the blank using a logical guess, like the average (mean) of the other data points.

B. Handling Outliers

Outliers are extreme values that can skew your model. You can:
- Trim: Remove the top and bottom 1% of data.
- Winsorize: Instead of deleting the outliers, you cap them at a certain percentile (e.g., changing a 99th percentile value to the 95th percentile value).

C. Scaling and Normalization

Imagine comparing a stock price ($2,000) to a dividend yield (0.02). The model might think the price is "more important" just because the number is bigger. To fix this, we scale the data.
- Min-Max Scaling: Squishes all data into a range between 0 and 1.
- Standardization: Centers the data so the mean is 0 and the standard deviation is 1.

Mnemonic: S.O.M. (Scale, Outliers, Missing). Remember these three things when someone asks how to prep data!

4. Step 4: Model Building

Now we get to the "AI" part. We split our data into two groups:
1. Training Set: The data the model uses to "learn" patterns.
2. Test Set: New data used to see if the model actually works.

Common Mistake: Data Leakage. This happens if information from the test set "leaks" into the training set. It’s like a student seeing the answers to an exam before taking it—they’ll get a 100%, but they didn't actually learn anything!

Underfitting vs. Overfitting

- Underfitting: The model is too simple (High Bias). It misses the trend.
- Overfitting: The model is too complex and memorized the "noise" in the training data (High Variance). It performs great on training data but fails miserably on new, real-world data.

Key Takeaway: We want the "Goldilocks" model—not too simple, not too complex, but just right (low bias and low variance).

5. Step 5: Model Evaluation (Measuring Success)

How do we know if our model is any good? We use a Confusion Matrix for classification tasks (like "Buy" vs. "Sell").

Imagine a model predicting if a company will go bankrupt:
- True Positive (TP): Predicted bankruptcy, and it happened. (Correct!)
- True Negative (TN): Predicted no bankruptcy, and it didn't happen. (Correct!)
- False Positive (Type I Error): Predicted bankruptcy, but it stayed healthy. (False Alarm)
- False Negative (Type II Error): Predicted healthy, but it went bankrupt. (The most dangerous error!)

Key Metrics to Memorize:

1. Precision: Of all "Positive" predictions, how many were actually right?
$ Precision = \frac{TP}{TP + FP} $
2. Recall (Sensitivity): Of all the actual positive cases, how many did we catch?
$ Recall = \frac{TP}{TP + FN} $
3. Accuracy: The percentage of total correct predictions.
$ Accuracy = \frac{TP + TN}{Total} $
4. F1 Score: A harmonic mean of Precision and Recall. Use this when you have imbalanced data (e.g., 99% of companies don't go bankrupt, but 1% do).

Did you know? High accuracy can be deceiving! If 99% of companies are healthy, a "dumb" model that predicts "No Bankruptcy" for everyone will be 99% accurate but totally useless for finding the 1% that fail. This is why we use the F1 Score.

6. Visualizing Performance: ROC and AUC

For binary classification, we often use the Receiver Operating Characteristic (ROC) curve.
- It plots the True Positive Rate (Recall) against the False Positive Rate.
- The Area Under the Curve (AUC) tells us how good the model is at distinguishing between classes.
- An AUC of 1.0 is perfect. An AUC of 0.5 is no better than guessing (a coin flip).

Quick Summary & Final Review

- The Process: Concept -> Collect -> Prepare -> Build -> Evaluate.
- Wrangling: Fixing missing values and scaling data is vital for "fair" comparisons.
- Overfitting: The enemy of machine learning. It happens when a model is too flexible.
- Metrics: Don't just trust Accuracy. Look at Precision (avoiding false alarms) and Recall (not missing the target).
- Imbalanced Data: Always look for the F1 Score or AUC when the "Positive" event is rare.

Don't worry if the formulas for Precision and Recall seem confusing at first. Just remember: Precision is about being "precise" (not being wrong when you say 'Yes'), and Recall is about "recalling" or finding every instance that exists!

Quick check

Can you answer these now?

Open each question to check the key ideas from this chapter.

What are the five steps of the Big Data project lifecycle?

The lifecycle consists of (1) Conceptualizing the project, (2) Data collection, (3) Data preparation (wrangling), (4) Model building, and (5) Model evaluation.

What is the main difference between structured and unstructured data?

Structured data is organized in rows and columns (e.g., price data), while unstructured data is unorganized text or media (e.g., news articles) that requires Natural Language Processing (NLP).

How does imputation differ from exclusion when handling missing data?

Exclusion involves deleting rows with missing values, whereas imputation fills the gaps using logical estimates, such as the mean of existing data points.

What is the difference between trimming and winsorizing for outlier management?

Trimming removes extreme values from the dataset entirely, while winsorizing caps extreme values at a specific percentile (e.g., the 95th percentile).

What is the purpose of scaling or normalization in data preparation?

Scaling ensures that variables with different units or magnitudes (e.g., share price vs. yield) are treated fairly by the model and do not dominate based solely on their numerical size.

What is data leakage and why is it problematic?

Data leakage occurs when information from the test set is accidentally used to train the model, leading to overly optimistic and unrealistic performance results.

How do underfitting and overfitting relate to bias and variance?

Underfitting is characterized by high bias (too simple), while overfitting is characterized by high variance (too complex and capturing noise).

In a bankruptcy prediction model, what constitutes a False Negative (Type II error)?

A False Negative occurs when the model predicts a company is healthy, but it actually goes bankrupt.

What is the difference between Precision and Recall?

Precision measures the accuracy of positive predictions ($ \frac{TP}{TP + FP} $), while Recall measures the ability to find all actual positive cases ($ \frac{TP}{TP + FN} $).

Why is the F1 Score useful for imbalanced data?

The F1 Score is the harmonic mean of Precision and Recall, providing a better performance measure when one class is much rarer than the other.

What does the Area Under the Curve (AUC) signify in an ROC curve?

AUC measures the model's ability to distinguish between classes, where 1.0 is perfect and 0.5 is no better than random guessing.

Ready to test yourself?

Turn these notes into exam-style practice. Get unlimited AI questions on this topic with instant marking and explanations.

Practice This Topic

More CFA Level II chapters

* The content provided by thinka is generated by AI and may not always be accurate or up-to-date. Please use it as a supplementary resource and verify with official materials.

Put These Notes into Practice

Reading the notes is a great start. Now practise with unlimited AI-generated questions and get instant feedback. 100,000+ students are already improving their grades.

Start Practising Now View Pricing

Done reading? Test yourself with AI practice questions

Practice This Topic Now