Evaluating Success Rate
Let's suppose that you want 90% of your decisions to be correct. You keep track of 100 decisions and find that only 85 of them were correct. Should you worry?
One way to find out is to calculate a statistic called chi-square. You have 85 correct decisions and 15 incorrect decisions. You subtract each of those observed values from the corresponding expected value, square the difference, and divide by the expected value. For example, you expected 90 correct decisions and observed 85. Subtracting 85 from 90 leaves 5, the square of 5 is 25, and 25 divided by 90 is .28. Similarly, 10 minus 15 leaves -5, the square of -5 is 25, and 25 divided by 10 is 2.5.
To get chi-square you just add the results of those operations. So chi-square equals .28 + 2.50 = 2.78. The probability of obtaining a chi-square of 2.78 is greater than .05, which is the highest standard usually used for statistical significance. Consequently, we cannot reject the hypothesis that you are performing as well as you wanted to.
At this point you're probably wondering what value of chi- square you would need to obtain to reject the hypothesis. In the example, the value would be 5.99, if your significance criterion was a probability less than .05, and 9.21 if your significance criterion was a probability less than .01. However, the required values of chi-square vary, depending on what are called the number of degrees of freedom in the analysis. Before you try this method out, I strongly suggest you consult someone who has been trained to use this test.