Appendix A: Survey Methodology

Navigate this page:
Sampling and Interviewing
Questionnaire Design
Survey Administration
Response Rate
Weighting
Margin of Error 

The Third Lausanne Congress of World Evangelization, also known as Cape Town 2010, was a gathering of global evangelical Christian leaders in Cape Town, South Africa, from Oct. 16-25, 2010. The congress had about 4,500 participants from 198 countries or territories. The organizers of Cape Town 2010 – the Lausanne Committee for World Evangelization (LCWE) – aimed to bring together a “global parliament” of evangelical leaders. The selection of participants was largely decentralized, with LCWE’s international deputy directors working in each of 12 regions to select participants in approximate proportion to the estimated number of evangelicals in each country. Participants were selected to attend the congress using a variety of criteria, including age, gender and other considerations aimed at broadly representing the evangelical Christian Church.

The Pew Research Center’s Forum on Religion & Public Life (Pew Forum) conducted a survey of the leaders selected to participate in the congress using web and paper-and-pencil questionnaires between Aug. 16, 2010, and Dec 6, 2010.²⁴  The survey was administered by Princeton Survey Research Associates International (PSRAI). Nearly all questionnaires were completed by the end of the congress on Oct. 25, 2010; seven respondents completed the survey after that date. The survey was offered in nine languages, including English. A total of 2,196 leaders completed the survey.

In advance of the congress, the Lausanne Committee for World Evangelization provided a list, with email addresses, for 3,415 participants who had registered to attend the congress.²⁵  About seven-in-ten respondents (68%) completed the survey prior to arriving at the congress in Cape Town, while 32% completed the survey during or after the congress. Nearly all of the surveys completed before the congress (99%) were done online, while nearly all of the surveys completed during and after the congress (99%) were completed on paper.

An attempt was made to contact all leaders selected to attend the congress, and the majority of leaders were sent an email prior to the congress and asked to participate in the survey. A group of leaders coming from countries with more extensive restrictions on religious freedom were not contacted prior to the congress because of concerns that such contact might jeopardize their safety. Attempts were made to contact this group at the congress through a small card enclosed with registration materials and through announcements at the meeting. Approximately 49% of the total number of delegates attending the congress completed the survey.²⁶  The 2,196 leaders who completed the survey turned out to closely mirror the full set of leaders attending the congress in terms of region, gender, age and organization type (additional details are provided below). The survey data reported here are not weighted.

The questionnaire was designed by the Pew Forum. The survey was conducted in nine languages (English, Arabic, Chinese, French, German, Korean, Portuguese, Russian and Spanish). Questionnaires were translated and reviewed by PSRAI. Translations were also reviewed by consultants retained by the Pew Forum in order to ensure they accurately reflected the way Christians speak about religious issues in each of the nine languages. The comments
of the reviewers were independently evaluated and the translations were revised accordingly.

The online questionnaire included programming for skip patterns. Question order and response order was fixed for all questions in order to increase the comparability of the online and paper survey modes. This departs from typical practice in web surveys (and telephone surveys) where the order of response options and questions (especially in serial lists of questions) is sometimes randomized. The purpose of randomizing order is to control for potential primacy and recency effects in response option order and for serial item position effects in question order. However, the practical issues of conducting a survey in two modes and nine languages precluded the use of randomization. The fixed question order should be kept in mind when interpreting the survey results.

The Pew Forum conducted the survey with the cooperation of the Lausanne Committee for World Evangelization. It would not have been possible to contact those attending the congress without such support from LCWE. While members of the LCWE advisory group provided comments about the survey questionnaire prior to fielding, the Pew Forum had final authority and responsibility for the design of the questionnaire and retains sole responsibility for the analysis and interpretation of survey findings.

In the summer of 2010, the LCWE provided PSRAI with an initial database of delegates registered to attend the congress. On Aug. 16, 2010, PSRAI emailed an invitation to participate in the survey to the delegates in the database. The invitations to participate consisted of two components: (1) a letter from Douglas Birdsall, executive chair of the Lausanne Committee for World Evangelization, explaining the purpose of the survey, and (2) an invitation letter from the Pew Forum and PSRAI inviting participation and explaining procedures for completing the online survey. To protect respondents’ privacy, this letter did not identify respondents by name. The invitation letter emphasized PSRAI’s commitment to confidentiality by stating that the names of individual respondents would not be revealed and that data from the survey would be reported in aggregate form only. Respondents were also given the option of printing the survey from the website and returning it via fax or mail. PSRAI sent up to three reminder e-mails, prior to the congress, in an effort to increase participation.

The online survey was programmed by PSRAI and tested by both PSRAI and the Pew Forum. To ensure security, respondents were issued unique user names and directed to a fully secure Internet website to complete the survey. Respondents were able to log on and off the site to complete the survey at their convenience.

Prior to the start of the congress, 1,475 respondents completed the survey online. By the end of the field period, 1,480 respondents completed the online questionnaire.  

During the congress, two representatives from the Pew Forum worked on-site in Cape Town to reach participants who had not responded to the online survey or who had not been contacted prior to the congress because they were from “closed countries” (including China and several Middle Eastern countries) or because they had not yet registered when the preliminary list of names was provided to PSRAI.²⁷ A small card was inserted into all registration packets instructing those who had not yet completed the survey to visit a table staffed by the Pew Forum, where they received a paper copy of the survey. These paper copies also included the advance letter and the invitation letter, as well as instructions for completing the survey online. The table remained open for the duration of the conference, and participants were offered candy and drinks as a thank you for completing the survey. Announcements were made during plenary sessions, information appeared in the daily newspaper alerting participants to the survey, and Forum staff approached participants to solicit their participation. Participants were encouraged to complete the survey at the Pew Forum table or to take a copy to complete and return at their convenience. In total, approximately 2,500 paper questionnaires were distributed to conference attendees. Of these, 694 were returned complete at the congress.

The 2,196 survey respondents includes 1,480 who completed the survey online, 694 who completed the survey on paper during the congress and 22 who returned a hard-copy of the survey to PSRAI via fax or email.

During the congress, participants alerted Pew Forum staff to translation errors affecting one question in the printed Spanish questionnaire (Q53c) and one question in the printed Korean questionnaire (Q27b). It was also discovered that the formatting of four questions (Q24a-h, Q27a-d, Q28, and Q42) in the printed Arabic questionnaire garbled the scripting of words and made them illegible. All errors that were pointed out during the congress were corrected by PSRAI, and the Pew Forum reprinted the questionnaires for subsequent respondents. In the printed Arabic questionnaire, there was also an error in Q18h that was not identified until after the congress.

All respondents completing the survey online were required to log-in with a unique identification number. Additionally, the paper questionnaires asked respondents to fill in their unique identification number. These identification numbers were used to ensure that only one response from each participant was included in the final dataset.²⁸ Identification numbers were also compared against each of two databases of congress participants provided by LCWE – an initial database provided in summer 2010 and used to solicit survey participation from delegates registered to attend the congress, and a final database of congress attendees provided after the conclusion of the congress. After removing duplicate responses, 2,196 cases were retained in the final dataset.

Included in the 2,196 cases are 157 for whom participation in the congress is uncertain. This figure includes 112 people who completed the survey as a result of having been invited to participate on Aug. 16, 2010, because they were in the initial database of congress delegates but were not listed in the final database of congress attendees provided after the congress. It is uncertain whether they attended the congress or not. There were also 45 questionnaires completed at the congress that could not be matched to the final database. It is uncertain whether these respondents were actual congress participants, whether they were at the conference in some other capacity, or whether they provided incomplete or inaccurate identification numbers.

The table below summarizes the response rate for all participants who completed the survey based on the assumption that all eligible participants were successfully contacted and received an invitation to participate.
 

The data were not weighted. The table below compares all who either registered for the congress by July 17, 2010, or who were in the final database of those attending the congress with those who completed a survey. The composition of those who completed a survey closely resembles the overall composition of the group. As the table below shows, the demographic characteristics of the survey respondents are very similar in terms of gender, age, organization type and geographic region to the total group of Lausanne leaders at the congress.

The margin of sampling error for the full sample and various subgroups is shown below. In addition to sampling error, one should bear in mind that question wording and practical difficulties in conducting surveys can introduce error or bias in the findings of opinion polls.

Footnotes: 

24 The Pew survey was limited to official participants in the congress, not counting day participants, spouses, observers, staff, volunteers or the roughly 100,000 estimated virtual participants taking part in some or all of the events online. (return to text) 

25 Two participants did not have email addresses. (return to text) 

26 This figure assumes that all eligible leaders were successfully contacted and asked to participate in the survey. (return to text) 

27 The LCWE said that more than 200 Christian leaders from China were not permitted by the government to attend Cape Town 2010. See http://www.lausanne.org/news-releases/china-missing-from-global-table-as-cape-town-2010-congress-opened-sunday-in-south-africa.html. (return to text) 

28 A total of 58 questionnaires were judged to be duplicates, meaning that a participant completed the survey twice. In these cases, the survey that was the most complete was retained in the final dataset. If both questionnaires were equally complete, the one completed first was selected. (return to text) 

Photo credit: ©Ocean/Corbis