Evaluate and use evidence: Practice Questions

Consider the following situation regarding a new health supplement:

Source A: A blog post by a fitness influencer who claims the supplement helped them lose 5 kg in one week. The post includes a link to buy the product with a discount code.
Source B: A peer-reviewed study by a university biology department, which tested the supplement on 200 participants and found no significant weight loss compared to a placebo group.

Which of the following best assesses the reliability or credibility of these sources?

A researcher conducted a survey at an elite private university, asking students if they believe that a university degree is essential for career success. 85% of the 400 respondents agreed. The researcher concluded that the vast majority of young adults in the country believe that higher education is a prerequisite for professional achievement.

Which of the following identifies a flaw in the representativeness of the evidence?

A high school implemented a new mandatory mindfulness program for its students. After three months, a survey of 50 students from the school's high-achieving debate team showed that 90% felt less stressed. The principal concluded that the program was a success and should be rolled out across all schools in the district to improve student well-being.

Which of the following identifies the most significant flaw in the representativeness of the evidence?

A study conducted by a leading manufacturer of ergonomic office chairs claims that employees who sit in their premium 'Aero-Sit' model are 25% more productive than those using standard chairs. The study was based on observations of employees at the manufacturer's own headquarters, who were given the chairs for free and knew they were being monitored for productivity.

Which of the following factors most undermines the credibility of this evidence?

A recent study funded by the National Association of Confectioners found that children who consume a small amount of chocolate daily have higher test scores in mathematics than those who do not. The study concluded that the antioxidants in cocoa directly improve cognitive function in developing brains.

Which of the following, if true, most weakens the credibility of the study’s conclusion?

A local news report states: "In a recent survey of 15 residents found at the local dog park on a Tuesday morning, 100% of them said that the city should increase funding for public green spaces. Therefore, it is clear that the entire city population overwhelmingly supports spending more tax dollars on parks."

Identify one way in which the selectivity of this sample affects its representativeness regarding the city's general population.

Medical researchers conducted a survey at a local fitness center to determine the average number of hours adults sleep per night. They found that 80% of respondents sleep more than 7 hours. Based on this, they claimed that most adults in the city are well-rested.

Identify a significant weakness in the representativeness of this evidence.

A witness to a street robbery claimed the getaway car was a blue sedan. However, the witness was over 100 meters away and the streetlights were malfunctioning at the time of the incident. Additionally, the witness has a history of vision problems but was not wearing glasses that night.

Evaluate the credibility of this evidence based on the witness's 'ability to see'.

A local technology start-up recently released a survey claiming that their new productivity app saves users an average of five hours per week. The survey was sent via email to 500 early-adopters who had signed up for the app's beta testing phase. Out of these, 45 users replied. The start-up's CEO, who owns 80% of the company, stated in a press release: 'This data proves that our software is the most efficient tool on the market for any professional seeking to regain their free time.'

(a) Identify two factors that weaken the reliability of the CEO's statement.
(b) Evaluate the representativeness of the evidence, specifically focusing on selectivity and sample size.
(c) Explain why the CEO's claim that the app is the 'most efficient tool on the market' is an inference that goes beyond the evidence provided.

The government recently released data showing that crime rates in 'Area X' dropped by 15% after the installation of 50 new street lights. They concluded that 'lighting is the most effective tool for crime prevention.'

(a) Is the 15% drop a necessary or sufficient condition to prove that the lights caused the change? Explain.
(b) Identify a rash generalisation in the government's conclusion.
(c) Suggest one piece of further information that would be needed to corroborate the claim that the lights were the cause.