Chapter: Questionnaires and Bias
Welcome to your study notes on Questionnaires and Bias! Whenever researchers, businesses, or scientists want to find out what people think, eat, buy, or do, they often turn to questionnaires. However, asking questions the wrong way can lead to misleading or completely incorrect results. In this chapter, you will learn how to design fair, reliable questionnaires and identify the hidden traps of bias.
Don't worry if this seems like a lot of details at first. Once you learn a few simple rules, spotting bad survey questions and identifying bias will become second nature!
1. What is a Questionnaire?
A questionnaire is a structured set of written questions used to collect data from respondents. It is one of the most common tools for gathering primary data (data collected firsthand by the researcher).
Open vs. Closed Questions
Every question in a survey can generally be grouped into one of two main types:
1. Open Questions:
These allow the respondent to answer in their own words without any preset options.
• Example: "What is your favourite thing about school lunch?"
• Advantages: Provides detailed, rich, and unexpected insights; respondents are not forced into categories.
• Disadvantages: Difficult and time-consuming to categorise, code, and analyse statistically.
2. Closed Questions:
These provide a fixed set of answers for the respondent to choose from (such as tick boxes, multiple choice, or rating scales).
• Example: "How often do you visit the school canteen? [ ] Daily [ ] 2–3 times a week [ ] Once a week [ ] Never"
• Advantages: Quick and easy to answer; fast and straightforward to summarise and analyse numerically.
• Disadvantages: Limits respondents' choices; might miss an option someone actually wanted to choose.
Did you know? A special type of closed question is the Likert Scale. This asks people to rate their level of agreement, such as: Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree.
Key Takeaway: Use closed questions when you want clear, numerical data that is easy to graph, and open questions when you need deep opinions or exploratory feedback.
2. The Golden Rules of Questionnaire Design
In GCSE Statistics exams, you are frequently asked to look at a poor survey question, critique it, and rewrite it correctly. Keep these essential design rules in mind:
Rule 1: Avoid Leading (Biased) Questions
A question should never nudge the person towards a particular answer.
• Bad: "Don't you agree that football is the best sport in Northern Ireland?" (This pushes the respondent to agree).
• Good: "Which sport do you prefer watching the most?"
Rule 2: Provide Non-Overlapping (Mutually Exclusive) Response Boxes
Response boxes must never overlap, otherwise a person will not know which box to tick.
• Bad: Options: [ ] 0–5 [ ] 5–10 [ ] 10–15 (Where does a person with \(5\) or \(10\) tick?)
• Good: Options: [ ] 0–4 [ ] 5–9 [ ] 10–14 [ ] 15+
Rule 3: Ensure Options are Exhaustive (Cover All Possibilities)
Make sure there is a valid choice for everyone, including an "Other" box or a zero option where applicable.
• Bad: "What is your favourite pet? [ ] Dog [ ] Cat" (What if you own a rabbit or no pet?)
• Good: Provide common choices plus [ ] Other (please specify) and [ ] None.
Rule 4: Always Include a Specific Timeframe
Vague questions lead to unreliable data because people interpret words differently.
• Bad: "How often do you exercise? [ ] A lot [ ] Sometimes [ ] Never" (What does "a lot" mean?)
• Good: "In a typical week, how many hours do you spend exercising? [ ] 0–1 hours [ ] 2–3 hours [ ] 4–5 hours [ ] 6+ hours"
Rule 5: Keep Questions Simple, Clear, and Polite
• Avoid double negatives (e.g., "Do you not disagree that...").
• Avoid asking two questions at once (e.g., "Do you enjoy maths and science?" — what if you only like one?).
• Avoid asking overly personal or sensitive questions unless absolutely necessary and completely anonymous.
Memory Aid: The "T-O-N-E" Check
When reviewing a survey question in an exam, check for T-O-N-E:
• Timeframe: Is a clear period (e.g., per week, per month) stated?
• Overlap: Are the response boxes distinct with no overlaps?
• Neutral: Is the question fair and unbiased (not leading)?
• Exhaustive: Is there an option for everyone (including "None" or "Other")?
Key Takeaway: A great question is neutral, gives a clear timeframe, and provides tick boxes that are non-overlapping and exhaustive.
3. Pilot Surveys (Pre-Testing)
Before launching a full-scale survey to hundreds or thousands of people, researchers conduct a pilot survey.
What is a Pilot Survey?
A pilot survey is a small-scale trial run of the questionnaire conducted on a small sample of people.
Why is a Pilot Survey Important?
• Identifies confusing or ambiguous questions: If respondents misunderstand a question, it can be reworded before the main survey.
• Checks response options: Reveals if common options were missed in closed questions.
• Estimates time and cost: Helps find out how long the questionnaire takes to complete.
• Tests data processing: Ensures the collected data can be easily coded and analysed.
Key Takeaway: A pilot survey is like a dress rehearsal — it catches mistakes early so you don't waste time and money on a flawed investigation.
4. Methods of Collecting Data Using Questionnaires
There are several ways to administer a questionnaire. Each method has its own strengths and limitations:
1. Postal Questionnaires
• Pros: Inexpensive for covering large geographical areas; respondents can complete it in their own time; anonymous.
• Cons: Very low response rate; no one is there to explain confusing questions; requires a follow-up reminder.
2. Face-to-Face Interviews
• Pros: High response rate; the interviewer can clarify difficult questions; can assess visual reactions.
• Cons: Expensive and time-consuming; risk of interviewer bias (respondent gives answers to please the interviewer).
3. Telephone Surveys
• Pros: Faster than postal surveys; wider geographical reach than face-to-face interviews; cheaper than in-person visits.
• Cons: People may hang up (moderate response rate); limited call times; some groups do not answer unknown numbers.
4. Online / Internet Surveys
• Pros: Very cheap and fast; results are collected and tabulated automatically in real-time; can reach vast numbers of people.
• Cons: Excludes people without internet access or digital devices (e.g., some elderly populations); self-selection bias is common.
Key Takeaway: The choice of method depends on budget, time constraints, target population, and how complex the questions are.
5. Understanding and Identifying Bias
Bias is any systematic error that leads to an unfair or inaccurate representation of the population being studied.
Major Sources of Bias:
1. Sampling Bias:
Occurs when the sample chosen does not represent the whole population.
• Example: Surveying people outside a gym at \(7\text{ am}\) to find the average fitness level of the town. This over-represents fit and active people.
2. Non-Response Bias:
Occurs when the people who choose not to reply have different opinions or characteristics from those who do.
• Example: A customer satisfaction survey where only extremely angry or extremely delighted customers bother to reply.
3. Question (Measurement) Bias:
Caused by leading, loaded, or confusing wording in the questions themselves.
• Example: "Given the terrible traffic, should we build a bypass?"
4. Social Desirability / Response Bias:
Occurs when respondents do not answer honestly because they want to look good or give socially acceptable answers.
• Example: People over-reporting how much fruit they eat or under-reporting how much television they watch.
5. Interviewer Bias:
The interviewer’s tone, body language, gender, age, or presence influences the participant's response.
• Example: A teacher asking students directly: "Do you always do your homework on time?"
How to Minimise Bias:
• Use random sampling methods where every member of the population has an equal chance of selection.
• Keep questionnaires strictly anonymous and confidential.
• Use neutral, unbiased wording.
• Carry out a pilot survey to identify and correct potential problems before full distribution.
Key Takeaway: Bias causes your sample statistics to deviate unfairly from the true population value. Minimise bias through thoughtful question design, appropriate delivery methods, and fair sampling.
Quick Review: Common Mistakes to Avoid in Exams
• Mistake 1: Writing overlapping boxes like \(0 - 10\), \(10 - 20\), \(20 - 30\).
Correction: Always make boundaries distinct: \(0 - 9\), \(10 - 19\), \(20 - 29\), or write \(1 - 10\), \(11 - 20\).
• Mistake 2: Forgetting to include a timeframe in frequency questions.
Correction: Always specify "per day", "per week", or "per month".
• Mistake 3: Confusing a pilot survey with a regular sample.
Correction: Remember that a pilot survey is a small test run designed to fix problems with the survey itself.
• Mistake 4: Providing options with gaps where some people cannot answer.
Correction: Ensure all numbers are covered and include an "Other / None" option when listing categories.