Introduction: Understanding Relationships in Data

In the world of actuarial science, we rarely look at a single piece of data in isolation. Instead, we are often interested in how two things move together. For example, if the age of a policyholder increases, does the frequency of their insurance claims also increase? This is what we call correlation. In this chapter, we will explore three key ways to measure the strength and direction of the relationship between two variables (bivariate data): Pearson’s, Spearman’s, and Kendall’s correlation coefficients. Think of these as different "lenses" through which we can view the same data to see different types of connections.

1. Pearson’s Product-Moment Correlation Coefficient

Pearson’s correlation, usually denoted by the letter \(r\) for a sample or \(\rho\) for a population, is the most common measure of correlation. It measures the strength of the linear relationship between two continuous variables.

What does it tell us?

The value of \(r\) always lies between \(-1\) and \(+1\):
  • \(r = +1\): A perfect positive linear relationship (as one goes up, the other goes up in a straight line).
  • \(r = -1\): A perfect negative linear relationship (as one goes up, the other goes down in a straight line).
  • \(r = 0\): No linear relationship at all.

The Formula

For a sample of \(n\) pairs of data \((x_i, y_i)\), the formula is:

\(r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}}\)

You might also see it written using the "sums of squares" notation:

\(r = \frac{S_{xy}}{\sqrt{S_{xx} S_{yy}}}\)

Important Considerations

Pearson’s correlation is very sensitive. It assumes the relationship is a straight line and is heavily influenced by outliers (extreme values). If your data curves (like a "U" shape), Pearson might tell you there is no correlation even if a strong relationship exists!
Key Takeaway: Use Pearson when you expect a straight-line relationship and your data doesn't have extreme outliers.

2. Spearman’s Rank Correlation Coefficient

Sometimes, we don't care about the exact values, but rather the rank or order of the data. This is where Spearman’s rank correlation (\(r_s\) or \(\rho_s\)) comes in.

How it works

Instead of using the raw numbers (like \$1,250 and \$5,000), we rank them (1st, 2nd, 3rd, etc.). We then calculate the Pearson correlation on these ranks. If there are no tied ranks, we use this simplified formula:

\(r_s = 1 - \frac{6 \sum d_i^2}{n(n^2 - 1)}\)

Where \(d_i\) is the difference between the rank of \(x_i\) and the rank of \(y_i\).

Why use it?

Spearman’s correlation measures monotonic relationships. A relationship is monotonic if, as one variable increases, the other variable tends to increase (even if it's not in a straight line).
  • It is robust to outliers because ranks limit the influence of extreme values.
  • It can be used for ordinal data (data that has a natural order, like "low", "medium", "high").

Key Takeaway: Spearman is great for "curvy" but ever-increasing relationships or data with messy outliers.

3. Kendall’s Rank Correlation Coefficient (Tau)

Kendall’s Tau (\(\tau\)) is another rank-based measure, but it works a bit differently than Spearman’s. It looks at the direction of pairs of data points.

Concordant vs. Discordant Pairs

Imagine picking any two people from a dataset:
  • A pair is concordant if one person is ranked higher than the other in both variables. They "agree" on the direction.
  • A pair is discordant if one person is ranked higher in the first variable but lower in the second. They "disagree".
The formula is:

\(\tau = \frac{(\text{number of concordant pairs}) - (\text{number of discordant pairs})}{\frac{1}{2} n(n - 1)}\)

Why use Kendall’s?

While Spearman is more common, Kendall’s Tau is often preferred when:
  • The dataset is very small.
  • There are many tied ranks.
  • You want a measure that has better statistical properties for hypothesis testing in certain scenarios.

Key Takeaway: Kendall’s looks at "agreement" between pairs and is very useful for small, tie-heavy datasets.

4. Statistical Inference for Correlation

Calculating a number is just the start. As actuaries, we need to know: Is this correlation real, or just a result of random chance?

Hypothesis Testing

We usually test the null hypothesis:

\(H_0: \rho = 0\) (No correlation exists in the population)

Against the alternative:

\(H_1: \rho \neq 0\) (A correlation exists)

For Pearson’s correlation, if the data follows a bivariate normal distribution, we can use a \(t\)-test with \(n - 2\) degrees of freedom:

\(t = r \sqrt{\frac{n - 2}{1 - r^2}}\)

Note: Similar tests exist for Spearman and Kendall, often involving looking up critical values in tables or using software like R (as you will do in Paper B).

Summary and Comparison

Quick Comparison Table

Pearson: Linear relationship; sensitive to outliers; requires continuous data.
Spearman: Monotonic relationship; uses ranks; robust to outliers.
Kendall: Based on pair agreement; best for small samples or many ties.

Common Mistakes to Avoid
  • Correlation \(\neq\) Causation: Just because two things are correlated doesn't mean one causes the other. (e.g., Ice cream sales and shark attacks are correlated because of warm weather, not because ice cream attracts sharks!)
  • Misinterpreting \(r = 0\): For Pearson, \(r = 0\) only means there is no linear relationship. There could still be a very strong non-linear relationship.
  • Ignoring the scatterplot: Always look at a visual plot of your data before choosing a correlation measure!
Quick Review Questions
  1. Which measure would you use for a relationship that looks like a curve but always goes upward? (Answer: Spearman or Kendall)
  2. If \(r = -0.9\), what does this tell you about the relationship? (Answer: Strong negative linear relationship)
  3. Why is Spearman often preferred over Pearson for data with outliers? (Answer: Because ranks "neutralize" the impact of extreme values)

For more information on visualizing this data, see the chapter on "Summary statistics and exploratory data visualisation". To see how these variables relate in a predictive model, refer to the "Regression theory and applications" section.