Checklist
of Potential Problems with Surveys
(See
A. Sampling
Procedures
1. Is the sample a haphazard (nonprobability) sample or some variant of a probability (random) sample?
Examples of haphazard samples are: "person-on-the-street" interviews,
letters to the editor, call-in polls, "straw" polls, Literary Digest, etc. Problems are bias, nonrepresentativeness.
Probability samples give each individual
from the population an equal chance of being selected. They allow for
generalizability with some degree of sampling error.
2. What is the size of the sample? What is the "sampling error,"
or the "accuracy level" of the survey and how does this affect the
interpretation of the survey findings?
Smaller samples (especially less than about
600 respondents) begin to yield intolerably high levels (4% and higher) of sampling error – the error or inaccuracy
in being able to generalize from sample results to the population. For example, for a sample size of 600 and a
sampling error of + or - 4%, if we find that 50% of the respondents in the
sample prefer candidate X to candidate Y, this actually means that we are
relatively certain (there is a 95% probability) that between 46% and 54% of the
American public prefer candidate X to Y.
Also, sampling errors are larger for smaller subgroups (e.g., women vs.
men) of the survey. Of course, if
accuracy isn't all that important, higher levels of sampling error may be
tolerable.
3. Was the interviewing done face-to-face or
over the telephone? How does this affect
results? If a telephone interview, was random digit dialing used to select
respondents?
4. What was the "response rate" of the
survey--i.e., percentage of those selected who refused to participate? How can this affect the survey results?
5. Note: Sampling errors are just the "tip of the
iceberg" in terms of problems or errors with public opinion polls and
reporting response rates, sampling errors, etc. in articles on political polls
tends to give the reader a false sense of the accuracy of polling results, as
if such errors are the only ones we
need to know about and that most of the “error” in a survey can be estimated
with scientific precision. In fact, other problems associated with question
wording, question order and the interpretation of survey findings are often
more important than sampling errors. In fact, if the poll is done by a
reputable firm, the sampling procedure is probably one of the least important aspects of the survey to
know about.
B. Question Wording
1. Is the question "loaded" or biased
in some way? Does it "lead"
respondents to answer in a particular manner?
Does it present different sides of an issue fairly?
2. Is the question susceptible to social
desirability biases so that some answers might appear more socially acceptable
or "politically correct?"
3. Is the question clear and unambiguous, simple
and straightforward? Or are there
several issues at stake in an unnecessarily complicated question? And does the question require knowledge that
many people may not have, or use terms that some people might not
understand? If so, the question may be
"testing" familiarity and measuring "nonattitudes" rather than
soliciting real opinions.
4. Are responses affected by the context of the
question--i.e., previous questions, question order, and the like?
5. Other question wording effects (see Erikson
and Tedin, Ch 2): Are there likely to be framing effects? Are the arguments
balanced? Are multiple Likert items balanced?
C. Interpreting Survey
Results
1.
Is there any reason to think that the polling organization or
sponsor is distorting the results of the poll for its own benefit?
2.
Are there alternative interpretations or explanations for the
results, besides those being reported or intimated? Could differences in responses across groups,
over time, etc. be due to some other reason than those suggested in the
article?
3.
What are the goals of the analyst? Mere description, explanation, or
prediction?
4.
What "model" of polling and public opinion do
pollsters and reporters seem to have in mind in describing and interpreting the
results of a poll? Two typical types of interpretations of political polls of
candidate or policy support:
Public opinion as “elections”: Is the public opinion poll being interpreted
as a sort of "interim election"
or a "mandate from the people" that should be followed by the
nation's leaders (George Gallup's position)?
Are the results being used to predict political behavior or support
weeks and months from now? If so, the
political attitudes being measured must be salient, stable, and
"strong" so that the “snap-shot” picture provided by the public
opinion poll is not a serious distortion.
Public opinion as a “puzzle” that needs
further probing and explanation: Or is the poll being used to understand the sources and dynamics of
public opinion, which is acknowledged to be complex and ever-changing? If so, is it acknowledged that much of public
opinion is often subject to change, and is sometimes amorphous, somewhat weak
and passive, with only a minority mobilized pro or con? Is there an attempt to understand how public
opinion changes in response to events and how those changes produce trends in
the "climate" of public opinion?
Is there an attempt to document trends in public opinion over time, to
understand the origins of public opinion, or document and explain differences
in public opinion across different social, political, and information groups in
the population?
How do
politicians, journalists, social scientists and the public differ in the way
they are likely to interpret polls, based on their goals?
5.
Would using other methods in addition to, or in lieu of
surveys help us to overcome limitations of opinion polls?
·
Use depth interviews or focus groups to: delve beneath the
surface of superficial survey responses and understand how people arrive at their opinions in the first place?
·
Use lab experiments or survey experiments to disentangle
causes from effects in public opinion?
·
Use Q-methodology to understand the different meanings and
subjective frames of reference that people use to interpret terms and
questions?