Complete exploratory data analysis

CS1 – Actuarial Statistics · IFoA (Institute and Faculty of Actuaries) · 6 min read

Welcome to the World of Exploratory Data Analysis (EDA)!

Welcome! Before an actuary builds a complex model to predict the future, they first need to understand the present. Think of Exploratory Data Analysis (EDA) as being a detective. Before you solve the "crime" (the business problem), you need to look at all the clues (the data) to see what story they are telling. In this chapter, we will learn how to clean, visualize, and summarize data so we can make better decisions later. Don't worry if you find statistics a bit dry at first—we'll break it down using real-life scenarios!

1. Understanding Your Data: The Raw Ingredients

Before we can analyze data, we need to know what kind of data we have. Not all data is created equal! Just like you wouldn't use a thermometer to measure how much someone likes a movie, we use different tools for different data types.

Categorical (Qualitative) Data

This describes "qualities" or "labels." It’s data that fits into groups.
• Nominal Data: These are just names with no specific order. Example: Eye color (Blue, Brown, Green).
• Ordinal Data: These have a natural order. Example: Customer satisfaction (Low, Medium, High).

Numerical (Quantitative) Data

This describes "quantities" or "numbers" you can do math with.
• Discrete Data: Countable values, usually whole numbers. Example: The number of insurance claims a person makes (0, 1, 2...). You can't have 1.5 claims!
• Continuous Data: Measurements that can take any value in a range. Example: The exact time it takes to process a claim (3.452 days) or the height of a person.

Quick Tip: If you can ask "How many?", it’s likely discrete. If you can ask "How much?", it’s likely continuous.

2. Summary Statistics: Describing the "Center" (Location)

Summary statistics help us turn a giant pile of numbers into a few meaningful values. First, we look for the "middle" of the data.

The Mean (Arithmetic Average)

The sum of all values divided by the number of observations.
Formula: $ \bar{x} = \frac{\sum_{i=1}^{n} x_i}{n} $
Real-world Example: If you have 5 insurance claims of $100, $200, $150, $300, and $1000, the mean is $ (100+200+150+300+1000) / 5 = \$350 $.

The Median

The middle value when data is sorted from smallest to largest. If you have an even number of data points, it’s the average of the two middle numbers.
Why use it? The median is "robust." It isn't easily swayed by outliers (extreme values). In our example above, the $1000 claim makes the mean look high, but the median is just $200. It's often a better "typical" value for skewed data.

The Mode

The value that appears most often. In our claim example, the mode is actually not clear, but if we had two claims of $100, the mode would be $100.

Key Takeaway: Use the Mean for symmetric data and the Median when you have extreme values that might distort the average.

3. Summary Statistics: Describing the "Spread" (Dispersion)

Knowing the center isn't enough. We need to know if the data is all clustered together or spread far apart.

Variance and Standard Deviation

These measure the average distance of data points from the mean.
• Variance ($ s^2 $): Average of the squared differences from the mean.
• Standard Deviation ($ s $): The square root of the variance. We use this more often because it's in the same units as the original data (e.g., Dollars, not Dollars squared).
Formula for sample standard deviation: $ s = \sqrt{\frac{\sum (x_i - \bar{x})^2}{n-1}} $

Range and Interquartile Range (IQR)

• Range: Max value minus Min value. Very sensitive to outliers.
• IQR: The distance between the 75th percentile ($ Q_3 $) and the 25th percentile ($ Q_1 $). It covers the middle 50% of your data.
Formula: $ IQR = Q_3 - Q_1 $

Did you know? Actuaries care deeply about spread because it represents risk. High spread in claim costs means more uncertainty for the insurance company!

4. Describing the Shape: Skewness

Is your data symmetrical like a bell, or does it have a long "tail" on one side?

• Symmetric: Mean $\approx$ Median $\approx$ Mode. The left side is a mirror of the right.
• Positively Skewed (Right-Skewed): The "tail" points to the right (higher values). Mean > Median. Example: Individual wealth (a few billionaires pull the mean up).
• Negatively Skewed (Left-Skewed): The "tail" points to the left (lower values). Mean < Median. Example: Age at retirement (most people retire late, a few retire very early).

Memory Aid: The skewness is where the "tail" is. If the tail is on the Right, it's Positive skew.

5. Graphical EDA: Seeing is Believing

Sometimes a picture is worth a thousand numbers.

Histograms

Used for continuous data. It groups data into "bins" (ranges) and shows the frequency of each bin. It’s the best way to see the shape (skewness) of your data.

Boxplots (Box-and-Whisker Plots)

A boxplot summarizes data using 5 key numbers:
1. Minimum (excluding outliers)
2. Lower Quartile ($ Q_1 $)
3. Median ($ Q_2 $)
4. Upper Quartile ($ Q_3 $)
5. Maximum (excluding outliers)

Step-by-Step Boxplot Interpretation:
• The "box" shows the IQR (middle 50%).
• The "line" inside the box is the median.
• "Whiskers" extend to the min and max.
• Points outside the whiskers are usually outliers.

Scatter Plots

Used to see the relationship between two numerical variables. For example, does the number of miles driven relate to the number of accidents? If the points form a line moving up, there is a positive correlation.

6. Identifying Outliers

An outlier is an observation that is "weirdly" far from the rest. In CS1, we often use the 1.5 x IQR Rule to find them.

How to find outliers:
1. Calculate $ IQR = Q_3 - Q_1 $.
2. Calculate the "Lower Fence": $ Q_1 - (1.5 \times IQR) $.
3. Calculate the "Upper Fence": $ Q_3 + (1.5 \times IQR) $.
4. Any data point below the lower fence or above the upper fence is an outlier!

Common Mistake: Don't just delete outliers! They might be errors, but they could also be the most important data points (like a massive insurance catastrophe claim).

Quick Review Box

• Categorical: Labels/Groups. Numerical: Measurements.
• Mean: Sensitive to outliers. Median: Not sensitive (robust).
• Right Skew: Tail on the right, Mean > Median.
• Outlier Rule: Anything outside $ [Q_1 - 1.5IQR, Q_3 + 1.5IQR] $.
• Histogram: Shows distribution shape. Boxplot: Shows quartiles and outliers.

Final Encouragement

EDA is the foundation of everything you will do in Actuarial Statistics. If you can master the art of looking at data and spotting patterns or oddities, you're already halfway to becoming a great actuary. Don't worry if the formulas for variance or skewness look intimidating—with practice, they become second nature!

Quick check

Can you answer these now?

Open each question to check the key ideas from this chapter.

What is the primary purpose of Exploratory Data Analysis (EDA) in an actuarial context?

EDA is the process of cleaning, visualizing, and summarizing data to understand its patterns and "story" before building complex predictive models.

What is the difference between nominal and ordinal categorical data?

Nominal data consists of names or labels without a specific order, while ordinal data has a natural, meaningful ranking or order.

How do discrete and continuous numerical data differ?

Discrete data consists of countable, usually whole-number values, whereas continuous data involves measurements that can take any value within a range.

What is the formula for the arithmetic mean?

The mean is the sum of all observations divided by the total number of observations: $ \bar{x} = \frac{\sum_{i=1}^{n} x_i}{n} $.

Why is the median considered a "robust" statistic?

The median is robust because it is not easily influenced by outliers; it represents the middle value of the sorted data regardless of extreme values at the ends.

What is the definition of the mode in a dataset?

The mode is the value that appears most frequently in a set of data.

What is the relationship between variance and standard deviation?

Standard deviation ($ s $) is the square root of the variance ($ s^2 $), used because it returns the measure of spread to the same units as the original data.

How is the Interquartile Range (IQR) calculated and what does it represent?

$ IQR = Q_3 - Q_1 $. It measures the spread of the middle 50% of the data.

How do the mean and median compare in a positively (right) skewed distribution?

In a positively skewed distribution, the mean is typically greater than the median because the long tail of high values pulls the mean upward.

What is the primary use of a histogram?

A histogram is used for continuous data to show the frequency of data points within specific intervals (bins), allowing us to see the shape of the distribution.

Which five values make up the "5-number summary" shown in a boxplot?

The five values are: Minimum, Lower Quartile ($ Q_1 $), Median ($ Q_2 $), Upper Quartile ($ Q_3 $), and Maximum.

When is a scatter plot used in EDA?

A scatter plot is used to visualize the relationship and correlation between two different numerical variables.

What is the "1.5 x IQR Rule" for identifying outliers?

A data point is an outlier if it is below $ Q_1 - (1.5 \times IQR) $ or above $ Q_3 + (1.5 \times IQR) $.

Ready to test yourself?

Turn these notes into exam-style practice. Get unlimited AI questions on this topic with instant marking and explanations.

Practice This Topic

More CS1 – Actuarial Statistics chapters

Describe the purpose and function of data analysis

* The content provided by thinka is generated by AI and may not always be accurate or up-to-date. Please use it as a supplementary resource and verify with official materials.

Put These Notes into Practice

Reading the notes is a great start. Now practise with unlimited AI-generated questions and get instant feedback. 100,000+ students are already improving their grades.

Start Practising Now View Pricing

Done reading? Test yourself with AI practice questions

Practice This Topic Now