Inferential testing: sign test, significance and choice of test

Introduction to Inferential Testing

Welcome to one of the most important chapters in Research Methods! So far, you have learned how to collect data and describe it using averages (like the mean) or graphs. But how do we know if our results actually mean something? Does a small difference between two groups happen because of the Independent Variable, or was it just down to luck?

Inferential testing provides the mathematical tools to answer this. It allows us to "infer" (conclude) whether our results are statistically significant or if they just happened by chance. Don't worry if the math sounds scary—for the AQA exam, you only need to calculate one simple test (the Sign Test) and understand the logic behind the others.

1. Probability and Significance

In Psychology, we never say we have "proven" something 100%. Instead, we talk about probability (represented by the letter \(p\)). Probability is a measure of how likely it is that our results happened by chance.

The 5% Rule (\(p \le 0.05\))

The standard level of significance in Psychology is 5%, written as \(p \le 0.05\). This means that for researchers to accept their results as significant, there must be a 5% or lower probability that the results were caused by fluke or chance. If the probability is higher than 5%, we must accept the Null Hypothesis (which states there is no effect).

The 1% Rule (\(p \le 0.01\))

Sometimes, psychologists need to be even more certain—for example, when testing a new drug or conducting a study that might be controversial. In these cases, they use a 1% level of significance (\(p \le 0.01\)). This is a "more stringent" (stricter) level of proof.

Using Statistical Tables

To decide if a result is significant, we compare our Calculated Value (the result we get from the math) against a Critical Value found in a pre-printed statistical table. To find the right critical value, you need to know:
1. Whether the hypothesis is one-tailed (directional) or two-tailed (non-directional).
2. The number of participants (usually represented as \(N\) or Degrees of Freedom \(df\)).
3. The significance level (usually \(0.05\)).

2. Type I and Type II Errors

Because we deal with probability, we can sometimes make mistakes. Psychologists categorize these into two types:

Type I Error: The "False Positive"
This happens when we say a result is significant, but it actually happened by chance. We reject the null hypothesis when we should have accepted it.
Analogy: A smoke alarm going off when there is no fire.
*This is more likely to happen if the significance level is too lenient (e.g., \(10\%\) instead of \(5\%\)).*

Type II Error: The "False Negative"
This happens when we say a result is NOT significant, but there actually was an effect. We accept the null hypothesis when we should have rejected it.
Analogy: A smoke alarm failing to go off even though the kitchen is on fire.
*This is more likely to happen if the significance level is too strict (e.g., \(0.01\%\)).*

3. The Sign Test

The Sign Test is the only statistical test you may be asked to calculate in the exam. It is used when we are looking for a difference between two sets of data.

When to use the Sign Test:

1. You are looking for a difference (not a correlation).
2. You have used a Related Design (Repeated Measures or Matched Pairs).
3. The data is Nominal (organized into categories).

How to calculate the Sign Test (Step-by-Step):

Step 1: Subtract the scores of Condition B from Condition A.
Step 2: Record the sign of the difference. If the score went up, it's a plus (\(+\)). If it went down, it's a minus (\(-\)). If it stayed the same, ignore it and reduce your \(N\) (total participants) accordingly.
Step 3: Count the number of pluses and the number of minuses.
Step 4: The Calculated Value (\(S\)) is the smaller of the two totals. For example, if you have 8 pluses and 2 minuses, your \(S = 2\).
Step 5: Compare your \(S\) value to the Critical Value table provided in the exam.
Crucial Rule: For the Sign Test, the calculated value (\(S\)) must be EQUAL TO or LESS THAN the critical value to be significant.

4. Factors Affecting the Choice of Test

In the exam, you will often be given a scenario and asked: "Which statistical test should the researcher use and why?" (usually worth 3-4 marks). To answer this, you must identify three things:

1. The Level of Measurement

How was the data recorded?
• Nominal: Data is in separate categories (e.g., Number of people who passed or failed).
• Ordinal: Data can be put in order/ranked, but the gaps between scores aren't equal (e.g., Rating your happiness on a scale of 1-10).
• Interval: Data measured using fixed, public scales with equal units (e.g., Temperature in Celsius, time in seconds, or height in cm).

2. Experimental Design

• Related Design: Repeated Measures (same people in both conditions) or Matched Pairs.
• Unrelated Design: Independent Groups (different people in each condition).

3. The Aim of the Study

• Is the researcher looking for a difference between groups?
• Or are they looking for a correlation (relationship) between two co-variables?

5. The Choice of Test Table

You need to memorize which test fits which criteria. Use this "cheat sheet" to help you:

Quick Tip: If the question mentions "Interval" data, but the data is skewed or doesn't meet specific requirements, psychologists often "drop" down to an Ordinal test. However, for AQA exams, identify the highest level of measurement possible in the description.

Summary Checklist

Key Takeaways:
• \(p \le 0.05\) is the standard level of significance.
• Type I Error is a "False Positive"; Type II Error is a "False Negative".
• For the Sign Test, the calculated value (\(S\)) is the less frequent sign.
• To choose a test, you must know the Level of Measurement, Design, and Aim.
• Memorize the names of the 8 tests: Spearman’s rho, Pearson’s r, Wilcoxon, Mann-Whitney, related t-test, unrelated t-test, Chi-Squared, and the Sign Test.