Welcome to Bivariate Distributions
In single-variable (univariate) statistics, we typically look at one measurement at a time, such as the height of a student or the score on a test. But in the real world, things are often connected! What if we want to look at two variables together, like the number of hours studied and the exam score achieved, or shoe size and height?
When we study two random variables at the same time, we call it a bivariate distribution. In this chapter of CCEA AS Further Mathematics, we will explore discrete bivariate distributions, how to describe the link between two variables, and how to calculate measures like covariance and correlation.
Don't worry if this seems like a lot of new notation at first! If you are comfortable finding probabilities, means, and variances for a single discrete variable, you already have all the foundational tools you need.
1. Joint Probability Mass Functions
Let \(X\) and \(Y\) be two discrete random variables. The joint probability mass function gives the probability that \(X\) takes a specific value \(x\) and \(Y\) takes a specific value \(y\) simultaneously.
We write this as:
\(P(X = x, Y = y) = p(x, y)\)
The Fundamental Rule of Joint Distributions
Just like with single variables where all probabilities must sum to \(1\), the sum of all joint probabilities over all possible values of \(x\) and \(y\) must equal \(1\):
\(\sum_x \sum_y P(X = x, Y = y) = 1\)
Representing Joint Distributions in a Grid
We often display joint probabilities in a two-way probability table. Each cell in the grid represents the probability \(P(X = x, Y = y)\) for a particular combination of \(x\) and \(y\).
Example:
Suppose a game involves rolling a die to determine \(X \in \{1, 2\}\) and flipping a coin to determine \(Y \in \{0, 1\}\). If the joint distribution is given by a formula such as \(P(X = x, Y = y) = k(x + y)\), we find \(k\) by adding all possible values and setting the total to \(1\):
For \((x, y) = (1, 0)\): \(P = k(1 + 0) = 1k\)
For \((x, y) = (1, 1)\): \(P = k(1 + 1) = 2k\)
For \((x, y) = (2, 0)\): \(P = k(2 + 0) = 2k\)
For \((x, y) = (2, 1)\): \(P = k(2 + 1) = 3k\)
Sum: \(1k + 2k + 2k + 3k = 8k = 1 \implies k = \frac{1}{8}\).
Key Takeaway: The joint probability is simply the probability of a specific pair \((x, y)\) occurring together. All cells in your probability grid must always add up to \(1\).
2. Marginal Probability Distributions
Sometimes we have a joint distribution table for \(X\) and \(Y\), but we only want to find the individual distribution of \(X\) (ignoring \(Y\)), or the individual distribution of \(Y\) (ignoring \(X\)). These individual distributions are called marginal distributions.
Why are they called "marginal"? Because in a table, you calculate them by adding up the rows and columns and writing the totals in the margins!
Formulas for Marginal Distributions
To find the marginal distribution of \(X\), sum over all possible values of \(Y\):
\(P(X = x) = p_X(x) = \sum_y P(X = x, Y = y)\)
To find the marginal distribution of \(Y\), sum over all possible values of \(X\):
\(P(Y = y) = p_Y(y) = \sum_x P(X = x, Y = y)\)
Calculating Marginal Means and Variances
Once you have the marginal distribution for \(X\) or \(Y\), you can find its expected value and variance using standard single-variable formulas:
\(E[X] = \mu_X = \sum x P(X = x)\)
\(E[X^2] = \sum x^2 P(X = x)\)
\(Var(X) = \sigma_X^2 = E[X^2] - (E[X])^2\)
The exact same method applies for \(E[Y]\) and \(Var(Y)\).
Key Takeaway: Summing across a row gives the marginal probability for that row's variable value; summing down a column gives the marginal probability for that column's variable value.
3. Conditional Probability Distributions
What happens to the distribution of \(X\) if we are given information about \(Y\)? This is a conditional distribution.
Think of it like filtering a dataset. If you already know that \(Y = 2\), you restrict your attention only to the column or row where \(Y = 2\).
Formulas for Conditional Distributions
The probability that \(X = x\) given that \(Y = y\) has occurred is:
\(P(X = x \mid Y = y) = \frac{P(X = x, Y = y)}{P(Y = y)}\)
Similarly, the probability that \(Y = y\) given that \(X = x\) has occurred is:
\(P(Y = y \mid X = x) = \frac{P(X = x, Y = y)}{P(X = x)}\)
Conditional Expectation and Variance
A conditional distribution is a valid probability distribution in its own right. Its probabilities sum to \(1\):
\(\sum_x P(X = x \mid Y = y) = 1\)
You can calculate the conditional expectation \(E[X \mid Y = y]\) just like any standard expectation, using the conditional probabilities:
\(E[X \mid Y = y] = \sum_x x \cdot P(X = x \mid Y = y)\)
\(E[X^2 \mid Y = y] = \sum_x x^2 \cdot P(X = x \mid Y = y)\)
\(Var(X \mid Y = y) = E[X^2 \mid Y = y] - (E[X \mid Y = y])^2\)
Key Takeaway: To find a conditional distribution, divide the joint probability by the marginal probability of the given condition. Don't forget to check that your conditional probabilities add up to \(1\)!
4. Independence of Random Variables
Two random variables \(X\) and \(Y\) are defined to be independent if the outcome of one does not affect the outcome of the other.
The Test for Independence
\(X\) and \(Y\) are independent if and only if:
\(P(X = x, Y = y) = P(X = x) \times P(Y = y)\)
for every single pair of values \((x, y)\).
Alternatively, in terms of conditional probabilities:
\(P(X = x \mid Y = y) = P(X = x)\) for all \(x, y\).
Common Mistake Alert: To prove independence, the multiplication rule must hold for every cell in the table. To prove that \(X\) and \(Y\) are not independent, you only need to find one counterexample pair \((x, y)\) where \(P(X = x, Y = y) \neq P(X = x)P(Y = y)\).
5. Expectation of Functions of Two Variables: \(E[g(X, Y)]\)
To find the expected value of any combined expression \(g(X, Y)\), multiply the value of \(g(x, y)\) by the joint probability for every cell in the distribution and add them all together:
\(E[g(X, Y)] = \sum_x \sum_y g(x, y) P(X = x, Y = y)\)
The Product Expectation \(E[XY]\)
A crucial special case is the expectation of the product \(XY\):
\(E[XY] = \sum_x \sum_y (x \cdot y) \cdot P(X = x, Y = y)\)
Quick Trick: Any term where \(x = 0\) or \(y = 0\) will have \(xy = 0\), so its product is zero and it does not contribute to the sum for \(E[XY]\)!
Linearity of Expectation
Expectation is always linear, regardless of whether \(X\) and \(Y\) are independent:
\(E[aX + bY + c] = aE[X] + bE[Y] + c\)
6. Covariance and Correlation
Covariance: \(Cov(X, Y)\)
Covariance measures the joint variability of two random variables. It tells us the direction of their linear relationship:
\(Cov(X, Y) = E[(X - \mu_X)(Y - \mu_Y)]\)
In practice, we almost always use the working formula for calculations:
\(Cov(X, Y) = E[XY] - E[X]E[Y]\)
Interpretation of Covariance:
- If \(Cov(X, Y) > 0\): \(X\) and \(Y\) tend to increase together (positive relationship).
- If \(Cov(X, Y) < 0\): As one variable increases, the other tends to decrease (negative relationship).
- If \(Cov(X, Y) = 0\): There is no linear association between \(X\) and \(Y\).
Important Property for Independent Variables
If \(X\) and \(Y\) are independent, then \(E[XY] = E[X]E[Y]\), which means:
\(Cov(X, Y) = 0\)
Caution: The reverse is not always true! A covariance of zero means there is no linear relationship, but there could still be a non-linear relationship between the variables.
The Product Moment Correlation Coefficient (\(\rho\))
Covariance depends on the units of measurement (e.g., cm vs metres). To get a standardized measure that does not depend on units, we calculate the population correlation coefficient, denoted by the Greek letter \(\rho\) (rho):
\(\rho = \frac{Cov(X, Y)}{\sigma_X \sigma_Y} = \frac{Cov(X, Y)}{\sqrt{Var(X)Var(Y)}}\)
Key Properties of \(\rho\):
- The value of \(\rho\) is always between \(-1\) and \(+1\): \(-1 \le \rho \le 1\).
- \(\rho = +1\) means a perfect positive linear relationship.
- \(\rho = -1\) means a perfect negative linear relationship.
- \(\rho = 0\) means no linear relationship.
7. Linear Combinations and Variance: \(Var(aX + bY)\)
When working with combinations of two random variables, their variances combine according to the following essential formula:
\(Var(aX + bY) = a^2Var(X) + b^2Var(Y) + 2ab\,Cov(X, Y)\)
Special Cases to Remember:
1. Difference between variables:
\(Var(aX - bY) = a^2Var(X) + b^2Var(Y) - 2ab\,Cov(X, Y)\)
2. Independent variables (where \(Cov(X, Y) = 0\)):
\(Var(aX + bY) = a^2Var(X) + b^2Var(Y)\)
\(Var(aX - bY) = a^2Var(X) + b^2Var(Y)\)
Notice that when variables are independent, we still ADD the variances even when taking the difference!
Step-by-Step Problem Solving Guide
When tackling a typical bivariate distribution question in an exam, follow these systematic steps:
Step 1: Check the total probability
Ensure \(\sum \sum P(X=x, Y=y) = 1\). If there is an unknown constant \(k\), use this to solve for \(k\).
Step 2: Find the marginal distributions
Sum across columns to find \(P(X = x)\) and sum across rows to find \(P(Y = y)\).
Step 3: Calculate individual expectations and variances
Find \(E[X]\), \(E[X^2]\), \(Var(X)\), \(E[Y]\), \(E[Y^2]\), and \(Var(Y)\).
Step 4: Find the joint expectation
Compute \(E[XY] = \sum \sum xy P(X=x, Y=y)\).
Step 5: Compute Covariance and Correlation
Use \(Cov(X, Y) = E[XY] - E[X]E[Y]\), then \(\rho = \frac{Cov(X, Y)}{\sqrt{Var(X)Var(Y)}}\).
Quick Review of Essential Formulas
Joint Total: \(\sum_x \sum_y P(X=x, Y=y) = 1\)
Marginals: \(P(X=x) = \sum_y P(X=x, Y=y)\), \quad \(P(Y=y) = \sum_x P(X=x, Y=y)\)
Conditionals: \(P(X=x \mid Y=y) = \frac{P(X=x, Y=y)}{P(Y=y)}\)
Independence Check: \(P(X=x, Y=y) = P(X=x)P(Y=y)\) for all \(x, y\)
Covariance: \(Cov(X, Y) = E[XY] - E[X]E[Y]\)
Correlation: \(\rho = \frac{Cov(X, Y)}{\sigma_X \sigma_Y}\)
Combined Variance: \(Var(aX + bY) = a^2Var(X) + b^2Var(Y) + 2ab\,Cov(X, Y)\)