Research, evaluation, analysis

The Satisfactory Satisfaction Survey
The spread of quality management has increased interest in monitoring and evaluating customer and employee satisfaction. Typically this involves soliciting customers' and employees' opinions by questionnaire. If you've thought about sending out customer satisfaction questionnaires, here are some suggestions that may increase the value of the information you get from your questionnaires.

1. The questionnaire is a first step, not a last step. Many organizations are content to assess satisfaction by sending out a questionnaire to a sample of customers, more or less asking them "How satisfied are you with our service/product? - Very satisfied-satisfied-unsatisfied-very unsatisfied". Of course, if 90% of the questionnaires are returned, and 90% of the people returning them report that they are very unsatisfied, you have some information on your hands that may be useful. If, as is more likely, questionnaires are returned by 30% of the people, and most of them report that they are either satisfied or very satisfied, you can't be too sure what that result means.
       The main problem is that you don't know why the other 70% of the people didn't return their questionnaires. Perhaps they were just too fed up with your products or your services to be bothered to do you the favour of returning the questionnaire. You can have the same problem in reverse. If questionnaires are returned by 30% of the people, and most of them are unhappy with you, your services, and your products, it is possible that the contented people didn't see a need to return your questionnaire.
       I recently analyzed the results of a customer satisfaction survey in which customers' age seemed to be negatively related to satisfaction with services. However, age was also negatively related to response rate. As customers' age increased, they were significantly more likely to return the questionnaire. If the younger customers tended to have less time in which to do things like filling out questionnaires, perhaps only the less contented of them were motivated enough to reply. In other words, the differences between younger and older customers in the survey sample may not have reflected differences between older and younger customers in the entire population of customers.
       To deal with this problem you compare your survey results to other measures of customer satisfaction. It is a general principle of measurement theory that conclusions cannot be effectively drawn from single measures. Combining your questionnaire results with information like requests for service, discontinuation of service, sales, or returns will give you a much better idea of satisfaction than will a questionnaire alone. Using consultation or focus groups to explore issues raised by your survey results can also be effective.
       A questionnaire is most effective as a way of explaining the way your customers behave. If your questionnaire results conflict with other measures of your customers' behaviour, there's something wrong with your questionnaire.

2. Opinions are only opinions. It is not uncommon for people's reported opinions, and even their reports of their own behaviour, to be inconsistent with the way they actually think or behave. For one thing, people often don't have very clear opinions, or very good ideas of how they typically behave. For another thing, they often don't care about how accurate their opinions are. As Gabriel Chanan has observed, people dispense opinions the same way they dispense matches.
       Validating your survey results with information obtained in other ways will reduce the danger of your questionnaire results leading you into incorrect decisions. In general, striking questionnaire results are only important if they are confirmed by other observations.

3. Ask more than a couple of questions. Responses to any single attitude item are influenced by many factors in addition to the attitude being assessed. Often the attitude will be less influential than these other factors in determining the response to the question.
       For example, if you evaluate employees' satisfaction with their jobs by asking them only "How satisfied are you with your job?", factors other than job satisfaction may affect the results. For example, employees of a charitable organization may be reluctant to express any opinion which may reflect negatively on an organization which helps the needy. During a recession, many employees would report being satisfied with any job. Employees with low incomes generally report higher job satisfaction than employees with high incomes, supposedly because they have more need to justify to themselves why they stay in their less remunerative jobs. These and many other factors can prevent you from finding out what factors are affecting, positively or negatively, the efficiency of employees.
       Similar factors will affect surveys of people who are using your services or buying your products. Clients given the opportunity to rate a family physician's services are unlikely to consider it an opportunity to castigate someone who has been healing them. On the other hand, customers given the opportunity to rate a product which is still under warranty, may look on it as an opportunity to justify any claims they may have to make under the warranty.
       The simple solution is to use a lot of items which deal with questions of fact. Asking people how much they agree with the statement, "My new Wonder Widget is useful" is less informative than asking them several items about whether they use it for specific purposes. Ask about what people do with your products or services, rather than about how people feel in general.
       Feelings are important, but a good survey designer will often not mention important issues explicitly in a survey questionnaire. A good questionnaire will elicit important information regardless of whether it's mentioned. This is an especially useful approach if you would have to use leading questions to ask about important issues.
       You may be concerned that people will be reluctant to complete a long questionnaire, or will complete it quickly and carelessly. However, people will carefully complete a long questionnaire if they believe that it is in their interest to complete it. They will not fill out a short questionnaire if they do not believe that it is in their interest to complete it. If you want people to complete your questionnaire, a number of simple techniques can be used to improve response rate.

4. The answers are more important than the questions. Questionnaires are often flawed by confusing wording of the questions. A common mistake is what is known as the double-barrelled question.
       For example, it is not uncommon to find people being asked to state how much they agree with statements like "Our service is friendly and efficient." Such a statement asks about two characteristics of service -- friendliness and efficiency -- and it would be best to ask for two ratings, one of friendliness and one of efficiency.
       If you don't split a statement like that into two statements, customers will be unsure how to answer. When they are unsure how to answer, they tend to answer in ways which are not representative of their real opinions. For example, they may make a "safe" response.
       The answers to questionnaire items may also be contaminated by customers' feelings about how socially desirable their response is, or by other factors which may distort their expression of their opinions. Using several items is one way of dealing with this problem.
       Finally, the ideal questionnaire item is one with which half the customers agree and half the customers disagree. A series of items on which 95% of the respondents give positive ratings may be a great thing to show to the boss, but results of this type actually provide less useful information than less positive results do. For statistical reasons, you won't be able to get much idea of the specific characteristics of your products or service which produce the high customer satisfaction.

5. Act. The worst deficiency of customer satisfaction studies is that often no one acts on them. Customer satisfaction may be studied so that products and services can be improved, or it may be studied so that pats on the back for a job perfectly done can be freely distributed throughout the organization.
       The first approach will always be the most productive. No organization is doing a flawless job, and a customer satisfaction study which fails to justify changes which will improve service is a waste of time and money.

 Related article | Another related article

The Satisfactory Satisfaction Survey © 1995, John FitzGerald

Home page  |  Decisionmakers' index | E-mail