The Trustworthiness of Survey Research

by Judith M. Tanur 
from the The Chronicle of Higher Education, May 25, 1994, pp. B1-B3.

Society relies on survey data for many important decisions, yet many people cannot tell a reliable survey from a questionable one. It is important for students and other non-specialists to learn to distinguish from the well done surveys and the slipshod and sometimes sensational ones. For example, in March 1993, Ross Perot announced that 97% of some 1.4 million respondents to questions printed on a postcard in TV Guide favored bigger cuts in government spending. When the same question was put to a random sample of people, the proportion of those in favor dropped to 67%. When the question was rewritten in a more-balanced fashion, the proportion dropped to 33%.

A  sophisticated user of survey data begins by asking questions:

• Who sponsored the survey--a government agency, a non-profit organization, an "interested party?"
  
• What procedures were used to determine the sample of people surveyed?
  
• How were the data collected--in person or by mail or telephone, with or without the assistance of computers?
  
• How were the questions worded and how were the options for responses arranged?

Selecting a sample

The main focus of a survey is not the responses themselves but to be able to use the sample to generalize the responses to a larger group or population. To make these generalizations accurately, the sample must be drawn using probability methods. Probability methods use chance instead of a researcher's judgment or a respondent's interest to decide the sample. For most cases, survey researchers have learned to be clever. They have devised ways to sample randomly generated telephone numbers or successively smaller geographic units (such as states within regions, towns within counties, blocks within villages, so on . .  down to individuals). 

Correctly drawn samples provide estimates of their own accuracy--a "margin of error." These estimates depend on the variability of the population, the size of the sample and the particular probability sampling method used. If sampling was not used, no estimation is possible and a reported margin of error should be viewed with extreme suspicion.

While government surveys use rigorous probability sampling, many surveys do not. If those who administer surveys are free to exercise their judgment about the sample, they may not be as rigorous as necessary. Why continue to make phone calls to reach a designated household, when the first call to another household could net a completed interview with little trouble?

Just as researchers should not select their respondents, neither should the decision be left to the respondents. Surveys in which respondents can nominate themselves abound--Zagat's Restaurant Surveys, the research in Shere Hite's book Women and Love, letters to Congressional representatives, as well as most call-in polls and the electoral process itself.

In a probability sample potential respondents can always choose not to participate. This can make a survey prone to bias. In order to avoid this researchers attempt to reduce non-response (and its resulting non-response bias) to a minimum by making repeated callbacks to research designated respondents and trying to convince reluctant individuals to participate. A good consumer of survey research must look for reports of response rates in surveys. Reputable surveys publish this information as a matter of course. When it is missing, be suspicious. Government surveys routinely achieve response rates of over 90%. Other well-done research is likely to get about 75%.

Question design

A large part of the literature in the field of survey research methodology deals with how to write questions and how to construct questionnaires.  Another part of the literature reports results of experiments that show that sometimes very minor changes in wording or in the order of alternatives given for responses can cause differences in responses. A good consumer of survey research should learn to look at the survey questions themselves in order to judge what the responses mean. Be suspicious of survey reporting that does not quote the questions verbatim.

Summary

Surveys are frequently used and useful tools, but they are dangerous if misused. Learn to be vigilant when creating or using survey research. When publishing results of research projects, press to include such information as response rates, margins of error, sampling methods, and exact wording of questions. Such information is important if readers and consumers are to judge the trustworthiness of the research.

How Outliers Can Explain Discrepancies!

by David Wheeler
Explaining the Discrepancies in Sex Surveys
Chronicle of Higher Education October 27, 1993, p. A9

Men report they have had about three times as many heterosexual partners as women say they have had. So large is the discrepancy that it has been difficult for people--from statisticians to sexologists--to explain it. Martina Morris reported in the September 30, 1993 issue of Nature that she has isolated the cause: a few men seem to have poor memories and are reporting very high numbers of female partners and throwing off the statistics.

Some researchers have used an old saw to explain the anomaly in past surveys: "Men like to boast, and women try to hide." However, according to Morris, accepting that maxim would mean that no one can be counted on to tell the truth about his or her sex life and that sex surveys are a waste of money.

Morris took a look at the latest comprehensive data available at that time--the University of Chicago's National Opinion Research Center added a few questions about sexual behavior to its annual poll, the 1989 General Social Survey. The data generated over a four-year period revealed the same discrepancy that is found in other sex studies. The 1,691 men who were surveyed reported a lifetime total of more than 20,500 female partners, while the 1,845 women surveyed reported only about 6,300 male partners, leaving 64% of the male sexual contacts unaccounted for.

When individuals who reported they had had more than 20 partners in their lifetimes were omitted, the ratio between male partners and female partners came much closer to a one-to one ratio--1.2 to 1. When those who had reported more than five partners in the past year were deleted from the analysis, the ratio become even more reliable. These individuals accounted for three percent of the surveyed group, suggesting that for 97% of the population sexual surveys are very accurate.

Morris notes that men who reported high numbers of partners were rounding off their figures by 5's, 10's, and even 25's. Does this mean men have trouble remembering? Morris suggests that as the number of partners increased and memories began to blur, gender stereotypes took hold: men may have exaggerated, while women may have turned to more modest estimates.

To avoid problems with asking survey respondents to remember, Morris suggests that researchers ask questions about shorter time frames and use questions that help people to remember. As an example, people can more accurately recall how many movies they have seen in the past year if they are also asked to think about which movies they saw.

 Idleman & Associates Turns Information into Knowledge