Peffley
Public Opinion/PS 473G
Extra Credit Written Assignment:

Critiquing a News Article’s Presentation of a Political Public Opinion Poll

Use the survey checklist below, as well as the discussion in class and ch.2 of Erickson and Tedin to evaluate and critique the reporting of results from a public opinion poll you find in a newspaper, magazine or some other NEWS source, where the public is likely to encounter a public opinion survey (excluding textbooks, periodicals like the Gallup Report, and polling web-sites, but including mainstream on-line news sources). Your critique is due in two weeks at the beginning of class and should be about 3-4 double-spaced typed pages with standard one-inch margins. The purpose of the assignment is to demonstrate your knowledge of the pitfalls of public opinion polls as well as to apply that knowledge in an astute way to critique the particular poll you find and the way it is reported. No additional outside reading is expected. Try to find an article that provides a more extensive discussion of a poll so that you’ll have more “fodder” to critique. Be sure to staple your poll article to your paper when you hand it in. Your grade will reflect not only the substance of your arguments but their clarity and organization, as well.

The paper is worth a total of 10 points to be added to your score on the midterm exam.

Checklist of Potential Problems with Surveys

(See Erickson and Tedin, APO, Ch 2)

A. Sampling Procedures

1. Is the sample a haphazard (nonprobability) sample or some variant of a probability (random) sample?

Examples of haphazard samples are: "person-on-the-street" interviews, letters to the editor, call-in polls, "straw" polls, Literary Digest, etc. Problems are bias, nonrepresentativeness.

Probability samples give each individual from the population an equal chance of being selected. They allow for generalizability with some degree of sampling error.

2. What is the size of the sample? What is the "sampling error," or the "accuracy level" of the survey and how does this affect the interpretation of the survey findings?

Smaller samples (especially less than about 600 respondents) begin to yield intolerably high levels (4% and higher) of sampling error – the error or inaccuracy in being able to generalize from sample results to the population. For example, for a sample size of 600 and a sampling error of + or - 4%, if we find that 50% of the respondents in the sample prefer candidate X to candidate Y, this actually means that we are relatively certain (there is a 95% probability) that between 46% and 54% of the American public prefer candidate X to Y. Also, sampling errors are larger for smaller subgroups (e.g., women vs. men) of the survey. Of course, if accuracy isn't all that important, higher levels of sampling error may be tolerable.

3. Was the interviewing done face-to-face or over the telephone? How does this affect results? If a telephone interview, was random digit dialing used to select respondents?

4. What was the "response rate" of the survey--i.e., percentage of those selected who refused to participate? How can this affect the survey results?

5. Note: Sampling errors are just the "tip of the iceberg" in terms of problems or errors with public opinion polls and reporting response rates, sampling errors, etc. in articles on political polls tends to give the reader a false sense of the accuracy of polling results, as if such errors are the only ones we need to know about and that most of the “error” in a survey can be estimated with scientific precision. In fact, other problems associated with question wording, question order and the interpretation of survey findings are often more important than sampling errors. In fact, if the poll is done by a reputable firm, the sampling procedure is probably one of the least important aspects of the survey to know about.

B. Question Wording

1. Is the question "loaded" or biased in some way? Does it "lead" respondents to answer in a particular manner? Does it present different sides of an issue fairly?

2. Is the question susceptible to social desirability biases so that some answers might appear more socially acceptable or "politically correct?"

3. Is the question clear and unambiguous, simple and straightforward? Or are there several issues at stake in an unnecessarily complicated question? And does the question require knowledge that many people may not have, or use terms that some people might not understand? If so, the question may be "testing" familiarity and measuring "nonattitudes" rather than soliciting real opinions.

4. Are responses affected by the context of the question--i.e., previous questions, question order, and the like?

5. Other question wording effects (see Erikson and Tedin, Ch 2): Are there likely to be framing effects? Are the arguments balanced? Are multiple Likert items balanced?

C. Interpreting Survey Results

1. Is there any reason to think that the polling organization or sponsor is distorting the results of the poll for its own benefit?

2. Are there alternative interpretations or explanations for the results, besides those being reported or intimated? Could differences in responses across groups, over time, etc. be due to some other reason than those suggested in the article?

3. What are the goals of the analyst? Mere description, explanation, or prediction?

4. What "model" of polling and public opinion do pollsters and reporters seem to have in mind in describing and interpreting the results of a poll? Two typical types of interpretations of political polls of candidate or policy support:

Public opinion as “elections”: Is the public opinion poll being interpreted as a sort of "interim election" or a "mandate from the people" that should be followed by the nation's leaders (George Gallup's position)? Are the results being used to predict political behavior or support weeks and months from now? If so, the political attitudes being measured must be salient, stable, and "strong" so that the “snap-shot” picture provided by the public opinion poll is not a serious distortion.

Public opinion as a “puzzle” that needs further probing and explanation: Or is the poll being used to understand the sources and dynamics of public opinion, which is acknowledged to be complex and ever-changing? If so, is it acknowledged that much of public opinion is often subject to change, and is sometimes amorphous, somewhat weak and passive, with only a minority mobilized pro or con? Is there an attempt to understand how public opinion changes in response to events and how those changes produce trends in the "climate" of public opinion? Is there an attempt to document trends in public opinion over time, to understand the origins of public opinion, or document and explain differences in public opinion across different social, political, and information groups in the population?

How do politicians, journalists, social scientists and the public differ in the way they are likely to interpret polls, based on their goals?

5. Would using other methods in addition to, or in lieu of surveys help us to overcome limitations of opinion polls?

· Use depth interviews or focus groups to: delve beneath the surface of superficial survey responses and understand how people arrive at their opinions in the first place?

· Use lab experiments or survey experiments to disentangle causes from effects in public opinion?