Appendix A: Survey Methodology
Results for this survey are based on telephone interviews conducted under the direction of Social Science Research Solutions (SSRS) among a national sample of 3,412 adults living in the continental United States, 18 years of age or older, from May 19-June 6, 2010* (2,393 respondents were interviewed on a landline telephone, and 1,019 were interviewed on a cell phone, including 444 who had no landline telephone). Interviews were conducted in English and Spanish.
All interviews were conducted using a Computer Assisted Telephone Interviewing (CATI) system, which ensures that questions were asked in the proper sequence with appropriate skip patterns. CATI also allows certain questions and certain answer choices to be rotated, eliminating potential biases from the sequencing of questions or answers.
How much do you know about religion?
And how do you compare with the average American? Take our short, 15-question quiz to find out.
For the landline sample, half of the time interviewers asked to speak with the youngest adult male currently at home and the other half of the time asked to speak with the youngest adult female currently at home. If no respondent of the initially requested gender was available, interviewers asked to speak with the youngest adult of the opposite gender who was currently at home. For the cell-phone sample, interviews were conducted with the person who answered the phone; interviewers verified that the person was an adult and could complete the call safely.
In an attempt to maximize survey response, unless an interview was completed or a callback scheduled for a respondent at a phone number in the sample, each number was contacted approximately seven times at varied times of day and days of the week. Cell-phone respondents also were offered a reimbursement of $5 to cover any costs of taking the call on their mobile phones.
The survey of the full national population used “random digit dial” (RDD) methodology. Samples of landline and cell phone exchanges were generated by Marketing Systems Group, a sister company of SSRS. The landline sample was “list-assisted,” meaning numbers were sampled from active “blocks” (area code plus three-digit exchange plus two-digit block number) that contained at least three residential directory listings; this is intended to exclude blocks dedicated for business or other nonresidential purposes.
The cell sample was not list-assisted but was drawn from systematic sampling of blocks dedicated to wireless phones and shared-service blocks with no directory-listed landline numbers.
The sample of 3,412 respondents included interviews with a nationally representative sample of 3,013 adults as well as an oversample of 399 people who are Jewish, Mormon, atheist or agnostic. One goal of the study was to attain sufficient numbers of interviews with members of these groups to permit reliable analysis of their religious knowledge. Oversampling was necessary because these groups account for a relatively small share of the overall population. Jews and Mormons each comprise roughly 1.7% of U.S. adults, according to the Pew Forum’s 2007 U.S. Religious Landscape Survey, while atheists and agnostics combined account for about 4% of the adult population, meaning that most surveys – even those based on large samples – do not include enough interviews with members of these groups to permit analysis of their views and characteristics.
Oversampling was accomplished by recontacting respondents from previous SSRS surveys. SSRS conducts nationally representative dual-frame (landline and cell phone) random-digit-dial surveys every week and asks respondents their religion; for this study, SSRS recontacted households it had reached in the preceding six months that contained at least one adult who reported being Jewish, Mormon or having no religion. Adults reached in those households were asked to confirm whether they were Jewish, Mormon, or atheist or agnostic and then the interview proceeded.
In total, the sample includes 212 interviews with Jewish respondents (56 interviewed as part of the main national sample and 156 interviewed as part of the oversampling), 215 interviews with Mormons (54 interviewed as part of the national sample and 161 as part of the oversampling), and 212 interviews with atheists and agnostics (130 interviewed as part of the national sample and 82 as part of the oversampling). The survey included a total of 74 self-described atheists, by themselves too small a group for reliable statistical analysis, so those individuals were aggregated with the 138 agnostics for this analysis.
The accompanying table shows the error attributable to sampling that would be expected at the 95% level of confidence for different groups in the survey.
A two-stage weighting design was applied to ensure an accurate representation of the national population.
The first stage, the base or design weight phase, included four steps. The first was a correction for the disproportionate oversamples and was created by dividing the percentage of each of the four oversampled groups (Jews, Mormons, atheists and agnostics) by their corresponding percentage in the main RDD sample. (Atheists and agnostics were treated as separate groups in this stage of the weighting process.) The second step was a correction for the unequal probabilities of selection that result from some households having more qualified adults than others; households reporting having a single adult received a weight of 1, while those with two or more adults received a weight of 2 (cell respondents were given the average weight of 1). Third, a weight of 0.5 was applied to all respondents who reported having both a landline and a cell phone, as they were twice as likely to be sampled as respondents who only had one phone type. Finally, the recontact sample was provided a propensity weight to account for the potential bias associated with recontacting (panel bias). These four corrections (sample type, household adults, dual frame and recontact propensity) were then multiplied together to arrive at a final base (design) weight.
The data were then put through a second weighting stage, a post-stratification sample balancing routine utilizing national estimates from the Census’ Current Population Survey Annual Social and Economic Supplement for age, race and ethnicity, gender, education and region of the country; from the decennial census for population density; and from the National Health Interview Survey for telephone usage (landline-only, cell-only or dual usage according to whether the respondent mostly uses the cell phone or uses the landline regularly).
Sample Disposition and Response Rate
The overall response rate for this study is 17.2% using formula “RR3” of the Standard Definitions of the American Association for Public Opinion Research (see http://www.aapor.org/AM/Template.cfm?Section=Standard_Definitions1 for detail on this calculation). Following is a full disposition of the sample selected for this survey:
* Correction added April 2011: Interviewing for the survey actually was completed on Friday, June 4, 2010. (return to text)Photo credit: Eric Swanson/Corbis