Research, evaluation, analysis

Who's Afraid of Small Samples?
Many people are reluctant to accept estimates based on small samples. Small samples are less accurate than large samples, but an important consideration that many people overlook is that we can estimate the accuracy of small samples.

Let's suppose that you want to know if there have been more men than women among 10,000 students who have enrolled at a school over the past 25 years. When you randomly select the records of 50 students you find that 70% are men. You have two options. You can randomly select more records to see if the trend holds or you can accept the result of your sample of 50 students. Which should you do?

Many people would be highly reluctant to accept these results, but from a statistical point of view you have little reason to select more records. Working out some simple formulas will show you that when you draw a sample of 50 people from a population of 10,000 and find that 35 of them are men, there is only one chance in a hundred that the actual percentage of men in the entire population is less than 53%. It's almost certain that there are more men than women at the school.

The example is artificial, but it demonstrates an important principle. That principle is that the adequacy of a sample is determined not by its size but by the decisions you can make by using it. Of course, the key here is random sampling. Estimates of sampling accuracy assume random sampling. If you can't sample randomly you can't estimate the accuracy of your estimates. As a fledgling researcher I was told a story about some researchers who selected every fifteenth name from a list they had been given of non-commissioned officers and other ranks at an army base and managed to select nearly every sergeant on the base. The list had been organized by platoon with the sergeant listed last in each platoon. I don't know if that story's true, but ever since I've used tables or generators of random numbers to draw names from lists.

Who's Afraid of Small Samples? © 1999, John FitzGerald
Home page | Decisionmakers' index | E-mail