Research, analysis, evaluation

In Defence of Opinion Surveys
Other articles posted here may leave you with the impression that I think opinion surveys are a waste of time. However, I actually think opinion surveys are like anything else – useful when used appropriately, but dangerous when used inappropriately. The widespread attitude that survey results are infallible is dangerous, particularly when we consider how often survey results are contradicted by reality. For example, political polling results are repeatedly contradicted by election results, but many people fail to draw from these contradictions the conclusion that survey results are fallible. Does anyone remember any pollster predicting the collapse of the Tories in the 1993 federal elections?

I recently had the opportunity to test the value of opinion surveys in a project for a client. This project included administering a quesionnaire of items designed to assess four dimensions of staff morale. Robert Karasek and Töres Theorell (Healthy work: Stress, productivity, and the reconstruction of working life. New York: Basic Books, 1990) have argued that other measures – of variables like staff turnover, absenteeism, and disability leave – are more useful for this purpose, so a study of these three variables was incorporated into the project.

These variables did turn out to be usefully correlated with morale. There are difficulties, however, in using them by themelves as measures of morale. First of all, they are subject to change for many reasons, and a change in them is difficult to attribute to any particular one of these reasons.

Staff turnover, absenteeism, disability leave, and similar variables also tend to be infrequent, so for various statistical reasons it is difficult to detect useful trends in these data.These measures also have little predictive value – knowing that absenteeism was low last year, for example, doesn't tell you much about what morale will be like this year.

A properly designed survey of morale, though, can pinpoint specific sources of satisfaction or dissatisfaction. My client's opinion survey was designed so that rating scales of four aspects of morale could be psychometrically derived and their reliability assessed. Analyses of changes in these scores and of differences between different parts of the organization produced findings similar to those produced by similar analyses of absenteeism, staff turnover, and disability leave.

The analyses of information other than opinions was therefore useful in establishing the validity of the opinion survey. However, once the validity of the opinion survey was established, the greater detail of the information it collected could then be used to elucidate the variation in the other variables.

For example, an increase in absenteeism may be attributable to any one of several factors, and even if it is related to changes in morale it may be more related to some aspects of morale than to others. A properly designed opinion survey can tell you which aspects of morale are the most important.

Opinion surveys of morale also have predictive value. Changes in absenteeism, staff turnover, and disability leave do not occur simultaneously with changes in morale, but some time after morale has started to change. Reasonably frequent surveys of morale can then help an organization avert unnecessary turnover and absenteeism.

These benefits of surveys of course extend beyond the study of morale. The keys to the success of opinion surveys, though, are psychometrically sound design and statistically sound analysis. Most political polls, or at least most of the ones whose results are made public, are inaccurate because they are not well designed – they assess attitudes with only one or two questions, for example – and because their results are naively interpreted.

Sound design and analysis usually lead to less spectacular conclusions than the ones derived from less well designed and analyzed surveys, but the sound conclusions will be infinitely more useful.

In Defence of the Opinion Survey © John FitzGerald, 1999

 Home page | Decisionmakers' index | E-mail