A warning on polls

Timothy McNulty | October 20, 2009

Here's something to file away when watching media coverage of next year's campaigns. Three leading Pa pollsters are getting concerned about the proliferation of polls out there, saying some of them are not meeting basic standards, including the wording of questions and who sponsored the surveys. (And no, they didn't name names.)

From a press release:

As academic survey researchers who conduct surveys for media outlets that are released to the public, we get many calls to comment about surveys produced by other pollsters. As we enter the 2010 midterm election cycle, the polls, especially the polls in Pennsylvania, are coming along at a record pace. Typically we receive calls from reporters and bloggers requesting comments about polling results and the methodologies used to produce those results. We are delighted to provide assistance when we can, but with the proliferation of polls using a variety of techniques to produce the results, it is impossible to make useful comments without the availability of certain basic information. There are many compelling other reasons for demanding transparency as well. Below we have included the standards for minimal disclosure for the public release of survey data as adopted and promulgated by the American Association of Public Opinion Research (APPOR), the nation's premiere organization representing survey researchers.  

According to AAPOR, it is unacceptable to release surveys without also disclosing certain kinds of information about how the poll was conducted - in other words, to meet standards of minimal disclosure. Over the past few years, we have given many talks and lectures about these disclosure standards, urging their use by the media. We understand that releases may not initially make available all the information in the standards but the disclosure information should be made available upon request or on a website. We continue to express optimism that deficient releases will not be given publicity or credence.

Chris Borick, Director of the Muhlenberg College Poll, Muhlenberg College
G. Terry Madonna, Director of the Center for Politics and Public Affairs and Franklin & Marshall College Poll, Franklin & Marshall College
Berwood Yost, Director of the Center for Opinion Research and Head Methodologist of the Franklin & Marshall College Poll, Franklin & Marshall College

AAPOR Code of Professional Ethics and Practices
Standards for Minimal Disclosure

Good professional practice imposes the obligation upon all public opinion researchers to include, in any report of research results, or to make available when that report is released, certain essential information about how the research was conducted. At a minimum, the following items should be disclosed:

1.      Who sponsored the survey, and who conducted it.
2.      The exact wording of questions asked, including the text of any preceding instruction or explanation to the interviewer or respondents that might reasonably be expected to affect the response.
3.      A definition of the population under study, and a description of the sampling frame used to identify this population.
4.      A description of the sample selection procedure, giving a clear indication of the method by which the respondents were selected by the researcher, or whether the respondents were entirely self-selected  
5.      Size of samples and, if applicable, completion rates and information on eligibility criteria and screening procedures.
6.      A discussion of the precision of the findings, including, if appropriate, estimates of sampling error, and a description of any weighting or estimating procedures used.
7.      Which results are based on parts of the sample, rather than on the total sample, and the size of such parts.
8.      Method, location, and dates of data collection.