A Question of Priorities
One of the uses of questionnaires is to collect peoples' opinions of priorities. Often people will be asked to rank priorities or to rate them on rating scales. The difficulties associated with using ranks were discussed last week, and the problem with rating scales is that in practice people tend to use them to assign high priorities to several items, so you end up with several items of roughly equal high priority. If you want to know the relative importance of these items to the people you surveyed you will be disappointed.
Last week I recommended just asking people to pick the single item of highest priority. Another option which can sometimes be even more effective is to use forced-choice items. These are items which present a pair of alternatives and ask the respondent to choose the one which is of higher priority to them. If you have a small number of items you can deal with all the priorities efficiently. For example, if you have four items you want to distinguish, you need only six forced-choice items to make all possible comparisons.
With larger numbers of items you can choose only the comparisons you are most interested in. You should have some type of control for response style bias, however. Some people just like to check the first item of a pair, or the second. As a control you might create two versions of your questionnaire, in each of which the choices are presented in opposite orders, and distribute each to half your sample.
I used forced-choice items in a survey earlier this year and obtained highly informative results. In general, when asked for priorities people will divide issues into two broad categories: the important and the unimportant. To rank the important priorities accurately you have to give your respondents the opportunity to provide more information than a set of rating scales is likely to let them provide.