Is the Difference Real?
This week we'll look at a few examples which demonstrate the value of testing for statistical significance:
Example 1. You poll 100 Americans and 100 Canadians about their political philosophies. You find that 54% of the Americans and 46% of the Canadians describe themselves as conservatives. Can you conclude that Americans are more likely to describe themselves as conservatives?
And the answer is: no. A statistical test shows that, if the citizens of the two countries do not on the whole differ in their self-descriptions, the likelihood of observing a difference of at least that size in two samples of 100 people from the two countries is too high for you to conclude that there is a real difference between the populations of the two countries (I used a lenient criterion for significance in all these examples).
Example 2. Refusing to give up, you poll 200 Americans and 200 Canadians about their political philosophies. You find that exactly 54% of the Americans (that is, 108 of 200) and exactly 46% of the Canadians describe themselves as conservatives. Can you now conclude that Americans are more likely to describe themselves as conservatives?
Again the answer is no. A statistical test shows that the probability of observing that difference between two samples even if there is no difference between the two populations is still too high.
Example 3. A polling company polls 1,100 people and asks for their choice of candidate in a forthcoming election. They find that 580 people (53%) support candidate A, and 520 people (47%) support candidate B. The polling company reports that 95% of the time its estimates should be accurate within 3 percentage points. Can you conclude that candidate A is more likely to win the election?
Once again, the answer is no. Leaving aside the question of whether the sample actually represents the population, the difference between the two percentages is simply not big enough to be considered significant by a statistical test.
If we hadn't tested for statistical significance, we might have been tempted to conclude that some of these differences were real. Testing for statistical significance helps us to avoid assuming a difference exists when it does not.