Appendix 3: Survey Methodology
Navigate this page:
Asian
Americans constitute a growing, but still rare population. According to the
2010 U.S. Census, Asian Americans constitute 5.6% of the U.S. population (and
5.5% of adults 18 years of age and older). The Asian-American population is
dispersed throughout the country, although about half live in the Western
region. Many Asian Americans are recent immigrants from multiple countries with
differing native tongues who likely have difficulty completing a public opinion
survey in English. Although the Asian-American population is quite diverse, the
six largest Asian subgroups—Chinese, Filipino, Indian, Vietnamese, Korean and
Japanese—represent 84.9% of all U.S. Asian adults.50
Despite
these challenges, the Pew Research Center 2012 Asian-American Survey was able
to complete interviews with 3,511 Asian-American adults 18 years of age and
older living in the United States from a probability sample consisting of
multiple sample sources that provided coverage for approximately 95% of the
Asian-American population. The survey was conducted in all 50 states, including
Alaska and Hawaii, and the District of Columbia. The survey was designed not
only to represent the overall Asian-American population but also the six
largest Asian subgroups (who each represent 0.4% to 1.3% of the adult U.S.
population). Interviews were completed with 728 Chinese, 504 Filipinos, 580
Asian Indians, 515 Japanese, 504 Koreans, 504 Vietnamese and 176 Asians of
other backgrounds.
Respondents
who identified as “Asian or Asian American, such as Chinese, Filipino, Indian,
Japanese, Korean or Vietnamese” were eligible to complete the survey interview,
including those who identified with more than one race and regardless of
Hispanic ethnicity. The question on racial identity also offered the following
categories: white, black or African American, American Indian or Alaska Native,
and Native Hawaiian or other Pacific Islander.
Classification
into U.S. Asian groups is based on self-identification of respondent’s
“specific Asian group.” Asian groups named in this open-ended question were “Chinese,
Filipino, Indian, Japanese, Korean, Vietnamese, or of some other Asian
background.” Respondents self-identified with more than 22 specific Asian
groups. Those who identified with more than one Asian group were classified
based on the group with which they identify most. Respondents who identified
their specific Asian group as Taiwanese or Chinese Taipei are classified as
Chinese-Americans for this report. See the questionnaire for more details.
Interviewing was conducted from Jan. 3 to
March 27, 2012 by Abt SRBI. Interviews were conducted in English as well as
Cantonese, Hindi, Japanese, Korean, Mandarin, Tagalog and Vietnamese. After
taking into account the complex sample design, the average margin of sampling
error for the 3,511 completed interviews with Asian-Americans is plus or minus
2.4 percentage points at the 95% level of confidence. The following table shows
the sample sizes and margins of error for different subgroups of Asians in the
survey:
Sample Design
The
sample design aimed to address the low incidence and diversity of the
Asian-American population while still achieving a probability-based sample by
employing multiple sampling frames, including landline and cell phone
random-digit-dial (RDD) samples and a sample of previously identified Asian
households. In addition, to complete a sufficient number of interviews with the
lowest incidence U.S. Asian groups, ethnic name based lists of “probable”
Filipino, Korean, Japanese, and Vietnamese households maintained by Experian
were used.51
RDD Geographic Strata
Since
the number of Asian Americans in national landline and cell RDD frames is quite
low, the landline and cell samples were geographically stratified to improve
the efficiency of the interviewing, where phone numbers in areas with higher
Asian-American incidence were selected at a higher rate than phone numbers in
areas with lower incidence. The differential selection rates were taken into
account in the weighting, which is described later in this section.
The
landline and RDD frames were divided into six stratum according to their
incidence of net Asian Americans and Asian American subgroups based on county-level
estimates from the U.S. Census Bureau’s American Community Survey (2007 to 2009
multi-year estimates). Counties were first grouped by the incidence of Asian
Americans as follows: Highest Density (35% and above), Higher Density stratum
(incidence 15% to 34.99%), High Density stratum (incidence 8% to 14.99%),
Medium Density stratum (incidence 4% to 7.99%), Low Density stratum (incidence
1.5% to 3.99%) and Lowest Density stratum (incidence under 1.5%).
Then,
an additional step was taken to see if counties should be reassigned to higher
density strata if they have a relatively high density of one or more of the
Asian subgroups. However, this step resulted in only minimal revisions to the
initial stratification. Only four counties in the U.S. have Japanese incidence
greater than 3% (they are all in Hawaii—Honolulu, Kauai, Maui, and Hawaii
counties); these were assigned to the Highest Density stratum. Only three
counties have Korean incidence greater than 3%; two were already in the Higher
Density stratum (Fairfax Co., VA and Bergen Co., NJ) and the third (Howard Co.,
MD) was reassigned to the Higher Density stratum due to the relatively high
incidence of Koreans. Two counties have Vietnamese incidence greater than 3%
(Santa Clara Co. and Orange Co., CA); both of these were already in the Highest
Density stratum. The Low, Medium, High, Higher, and Highest Density strata
cover 95% of all net Asians in the United States.
In
the Lowest Density Stratum, the incidence of Asian Americans is less than 1.5%.
In order to keep costs contained, counties in this stratum—which are home to
approximately 4.6% of all Asian Americans—were excluded from the landline and
cell RDD samples. Asian Americans living in the Lowest Density stratum were
still partially covered by the recontact and list samples.
Based
on previous Pew Research studies with English and Spanish interviewing, the
incidence of self-identified Asian Americans is approximately 1.5% in national
landline RDD samples and 3.1% in national cell RDD samples. To determine how
best to allocate interviews across the landline and cell frames, these
estimates along with additional analysis of Asians encountered in the screening
for Pew Research’s 2011 Muslim American Survey, were used to estimate the
expected incidences of Asian Americans in each strata in both the landline and
cell RDD frames. Based on these expected incidences, 60% of the RDD sample was
allocated to the cell RDD frame and 40% to the landline RDD frame. In the final
sample of completed RDD interviews, 58% were from the cell frame and 42% were
from the landline frame.
List Strata
To
reach the desired target of 500 completed interviews in each of the six largest
U.S. Asian subgroups—Chinese, Filipino, Indian, Japanese, Korean and
Vietnamese—the fresh RDD and recontact samples were supplemented with listed
samples based on ethnic names. Because of the size of the U.S. Chinese and
Asian Indian populations, list samples based on ethnic names were only needed
for the other four subgroups—Filipinos, Japanese, Korean and Vietnamese. In
total, the list samples were used to complete interviews with 226 Filipinos,
324 Japanese, 338 Koreans and 339 Vietnamese.
The
list samples were constructed from a commercial database of households where
someone in the household has a name commonly found in that Asian subgroup. The
lists were prepared by Experian, a commercial credit and market research firm
that collects and summarizes data from approximately 113,000,000 U.S.
households. The analysis of names was conducted by Ethnic Technologies, LLC, a
firm specializing in multicultural marketing lists, ethnic identification
software, and ethnic data appending services. According to Experian, the
analysis uses computer rules for first names, surnames, surname prefixes and
suffixes, and geographic criteria in a specific order to identify an
individual’s ethnicity and language preference.
Such
listed samples are fairly common in survey research, but when used alone they
do not constitute a representative sample of each Asian subgroup and the
probability that a given household belongs to one of these lists is not known. By
combining these lists with the landline RDD frame, however, the lists can be
used as components of a probability sample using statistical procedures that have
been developed to incorporate these types of listed samples into
probability-based surveys. The procedure implemented for this study involves
obtaining the entire list maintained by Experian for listed landline households
for Filipinos (n=140,163), Japanese (n=211,672), Koreans (n=164,710) and
Vietnamese (n=274,839). These four lists can be defined as four strata within
the entire landline RDD frame for the U.S. All telephone numbers drawn for the
geographic strata of the landline frame were compared to the entire Experian
lists for each of the four subgroups. Any numbers that appeared on
the Experian list frames were removed from the geographic RDD sample and were
available to be released only as part of the list strata. This
method makes it possible to determine the probability that any given Asian
American has of being sampled, regardless of whether he or she is included in
the Experian lists. It also permits estimation of the proportion of all Asian
Americans who are covered by the Experian lists, which in turn makes it
possible to give cases from the Experian samples an appropriate weight. The
list strata also provide some coverage of households in the Lowest Density
stratum since numbers in counties belonging to that stratum and appearing on
the Experian list were available to be sampled through the list samples.
In
some cases, the person identified in a list sample did not belong to the
nominal list group (i.e., either Filipino, Japanese, Korean, or Vietnamese).
When this occurred, the interview was terminated and coded as ineligible. This
protocol was motivated by considerations of weighting calculations and the
overall design effect from weighting.
Recontact Frame
In
order to obtain at least 500 interviews with each of the major Asian-American
subgroups, the national dual frame RDD samples were supplemented with a sample
of recontact cases. The recontact cases for this study came from several
sources. In order to maintain the probability-based nature of the study, only
recontact cases from prior national landline and cell RDD samples were used.
Furthermore, additional steps were taken to account for the probabilities of
selection in the original survey so that those could be reflected in the
weighting for this survey.
All
recontact numbers were checked against the Experian lists and the landline RDD and cell RDD samples and any numbers found were
removed from those samples and only available to be released as part of the
recontact sample.The recontact sample also provides some coverage of
households in the Lowest Density stratum.
Two-fifths
of the recontact interviews (604) came from self-identified Asian respondents
from prior national dual frame RDD surveys conducted by the Pew Research Center
or Abt SRBI since 2007. The remainder of the recontact sample came from the large,
dual frame RDD telephone screening effort implemented for the 2011 Pew Research
Muslim American Survey. All households identified as having a Muslim American
in 2011 were excluded from the Asian American Survey sample in order to avoid
the risk of context or conditioning effects. The 2011 Muslim American Survey
entailed screening over 41,000 households, and of these 3,585 households were
identified as likely having at least one Asian American adult and were not
interviewed in the Muslim American survey.52
Since it was known at the time that cases from the Muslim American Survey would
be used for the Asian American Survey, data was also collected on the specific
Asian languages encountered as well as the ethnic groups to which screened,
self-identified Asian Americans belong. The 3,585 recontact cases were of two
general types: self-identified Asian American households and Asian language
barrier households. The former group was comprised of 2,180 households in which
a respondent completed the screener, reported a religious affiliation other than
Muslim, and self-identified as Asian American to a question about racial
identification. The latter group was comprised of 1,405 households where the
screener was not completed, but the interviewer recorded that an Asian language
speaker was encountered. A total of 601 interviews were completed with
self-identified Asians and 280 interviews were completed with Asian language
barrier households from the 2011 Muslim American Survey.
Questionnaire
Development and Testing
The
questionnaire was developed by the Pew Research Center. The development of the
questionnaire was informed by feedback and advice from the panel of external
advisers. In order to improve the quality of the data, the English questionnaire
was piloted and then pretested with respondents using the Experian list sample.
Pilot Test and Pretest
For
the pilot test of selected questions from the survey, 100 interviews were
completed with Asian American adults sampled from the Experian lists. The
interviews were conducted October 6-11, 2011. Among households completing the
screener, the Asian incidence was 48%. The completion rate among qualified
Asians was 76%. The average length for Asian households was 18 minutes. Based
on the results of the pilot test, a number of changes were made to the
questionnaire and interviewer training procedures. There was no monetary
compensation for respondents for pilot interviews.
The
pretest of the full survey resulted in 15 completed interviews with Asian-American
adults sampled from the Experian lists. The interviews were conducted November
14, 2011. Among households completing the screener, the Asian American incidence
was 65%. The extended interview completion rate among qualified Asian Americans
was 68%. The average interview length for Asian households was 32 minutes.
Additional changes were made to the questionnaire and interviewer training
procedures based on the results of the pretest.
Translation of Questionnaire
Abt
SRBI used a professional translation service for all translations. The initial
translation used a three-step process of translation by a professional
translator, back translation to English by a second translator, followed by
proofreading and review for quality, consistency and relevance. As an
additional quality control, the translated questionnaires were reviewed by a
linguist from an independent translation service on behalf of the Pew Research
Center. The translated questionnaires were compared to the English source
document for accuracy of translation in the vernacular language. Discrepancies and differences of opinion about the most
appropriate translation were resolved using an iterative process, with the
original translator taking responsibility for reconciling all comments and
feedback into the final translation.
Survey
Administration
A
six call design was employed for both landline and cell phone numbers with no
callback limit for qualified Asian households. One attempt was made to convert
soft refusals in the landline sample with no refusal attempts for the cell
phone sample. Screening was conducted in English, with the exception of the
Korean and Vietnamese list samples which were conducted by bilingual (English
and Korean/Vietnamese) interviewers. Respondents speaking a foreign language
were asked what language they speak. Respondents identified as speaking
Mandarin, Cantonese, Korean, Vietnamese, Japanese, Hindi or Tagalog were
transferred to an appropriate foreign language interviewer or called back by an
interviewer who speaks the language if one was not immediately available.
All
Asian respondents were offered $20 for their participation. Interviews were
conducted in English (2,338 interviews), Cantonese (86 interviews), Mandarin
(130 interviews), Vietnamese (382 interviews), Tagalog (80 interviews),
Japanese (123 interviews), Korean (360 interviews) and Hindi (12 interviews).
This was achieved by deploying 262 English-speaking and 14 foreign
language-speaking interviewers. Only Korean and Vietnamese bilingual
interviewers were assigned to the Korean and Vietnamese Experian samples. Calls
were staggered over times of day and days of the week to maximize the chance of
making contact with potential respondents.
All
qualified callbacks and refusal landline cases which could be matched to an
address were sent a letter encouraging participation in the survey. All
language-barrier cases with an address were mailed letters translated into the
appropriate language (Chinese, Korean, Vietnamese, Hindi, Japanese or Tagalog).
A total of 1,131 letters were mailed.
Multilingual
interviewers on staff were utilized for the project as well as newly recruited
multilingual interviewers. New foreign language hires were first tested by an
accredited firm on their language proficiency then evaluated and scored before
being interviewed and hired by Abt SRBI. All multilingual interviewers first
went through the standard Abt SRBI training process that all interviewers
complete. Bilingual interviewers with more proficiency and interviewing
experience were given coaching/team leader roles and worked with the
interviewers in their language monitoring surveys, assisting in training and
debriefing.
The
screening effort yielded a response rate of 23% for the geographic landline RDD
sample, 13% for the cell RDD sample, 14 to 18% for the list samples and 54-79%
for the recontact samples, using the Response Rate 3 definition from the
American Association for Public Opinion Research. The recontact sample response
rates do not incorporate the response rates from the original surveys. Detailed
sample disposition reports and response rates for each sample sources are
provided in the supplemental tables.
The
completion rate for qualified Asian respondents was 73% for the geographic
landline RDD sample, 68% for the cell RDD sample, 84-92% for the list samples
and 81-86% for the recontact samples.
Weighting
Several
stages of statistical adjustment or weighting were needed to account for the
complex nature of the sample design. The weights account for numerous factors,
including (1) the geographic-based oversampling in the landline and cell RDD
frames, (2) the selection rates in the four list strata, (3) the original
probabilities of selection for the recontacts cases, (4) the presence of
unresolved numbers in the sample (unknown eligibility), (5) nonresponse to the
screener, (6) within household selection, (7) the overlap of the landline and
cell RDD frames, (8) nonresponse to the extended interview, and (9) the limited
coverage of the households in the Lowest Density stratum. Each of these
adjustments is detailed below.
The
first step in the weighting process was to account for the differential
probabilities of selection of phone numbers sampled for the study. As discussed
above, variation in the probabilities of selection came from several sources.
In the landline and cell RDD geographic samples, numbers in counties with
relatively high Asian American incidence were sampled at a higher rate than
numbers in counties with lower incidence. Also, the numbers in the four list
strata had selection probabilities higher than geographic RDD samples or the
recontact sample. The probability of selection adjustment is computed as
where
Nh is the number of telephone numbers in
the frame in stratum h, and nh is the number of telephone numbers
from stratum h in the released replicates. For the landline and cell RDD
geographic samples, stratum was defined by the cross-classification of frame
(landline or cell) and geographic stratum (Low, Medium, High, Higher, Highest).
For the four list samples, they each constituted their own stratum in the
national landline RDD frame. For most of the recontact sample, stratum was
defined as their sample (landline or cell) in the original survey. For the
recontact cases from the 2011 Pew Muslim American Survey, stratum was defined
by the cross-classification of frame (landline or cell) and geographic stratum
as specified for that survey.
At
the end of interviewing, each number dialed in the cell and landline geographic
samples plus the list strata was classified as eligible (working, residential
and not a minor’s phone), ineligible (non-working, business, or a minor’s
phone), or unknown eligibility (busy/no answer all attempts, fax/modem/computer
tone, answering machine/voice mail, or call blocked). For the recontact cases,
this information was available from archived disposition reports for the
original surveys. The base weights of the eligible cases were adjusted for the
fact that some of the “unknown eligibility” cases are likely to have been
eligible. This adjustment was performed by first computing the ratio of known
eligible cases in the stratum (Eh) to the sum of known eligible and
known ineligible cases in the stratum (Eh+Ih). This ratio
was then multiplied by the number of unknown eligibility cases in the stratum
(Uh) to yield an estimate for the number of eligible cases among
those with unknown eligibility (
). The adjustment is then computed as
The
next step was to adjust for nonresponse to the screener. The adjustment was
computed as
where 
is the total count of telephone numbers
in stratum h that are estimated to be residential and Sh is the number of
completed screener interviews in stratum h. Note that
as
suggested above.
Not
all of the Asian-American adults identified in the screener completed the
extended interview. An adjustment was performed by first computing the ratio of
known qualified Asian-American cases in the stratum (Ah) to the sum
of known qualified Asian American cases and screen-out (no Asian adults in
household) cases in the stratum (Ah+Bh). This ratio was
then multiplied by the number unscreened cases in the stratum (Ch)
to yield an estimate for the number of qualified Asian American cases among
those not screened (
). The adjustment is then computed as
The
weights were then adjusted to reflect the fact that only one eligible adult in
the household completed the screener interview. When landline numbers were
dialed, interviewers asked to speak with “the youngest male/female
[randomized], 18 years of
age and older, who is now at home.” When cell phone numbers were dialed,
the interviewer determined if the person answering the phone was 18 years of
age and older and, if so, proceeded with the beginning of the screener
interview. For both the landline and cell cases, if the initial screener
respondent was not Asian but reported an Asian adult in the household, then
ultimately an Asian adult became the final selected screener respondent. In
this situation, interviewers then asked to speak with the youngest male/female
[randomized], 18 years of age and older, who is Asian and is now at home.” In
order to ensure that Asian adults in households with multiple Asian adults are
not under-represented in the survey, an adjustment was computed to weight cases
up proportional to the number of Asian adults in the household. This adjustment
was truncated at 4 to avoid excessive variance in the weights.
The
base weights are the product of the aforementioned adjustments. The
distribution of the base weights was examined for any extreme values. The
distribution of base weight values for the recontact cases was noticeably
different from the distribution of the base weight for the balance of the
sample. The recontact cases had significantly larger base weights, on average,
owing to smaller sampling fractions relative to the geographic RDD samples and
list samples. Trimming of the base weights was, therefore, done separately for
the recontact cases and the balance of the sample. In both instances, the
threshold for trimming was the median + (6 x the interquartile range), which is
common for major surveys with complex sample designs.
The
next weighting step accounts for the overlap between the landline RDD frame and
the cellular RDD frame. The dual service (landline and cell-only) respondents
from the two frames were integrated in proportion to their effective sample
sizes. The first effective sample size was computed by filtering on the dual
service landline cases and computing the coefficient of variation (cv) of the
final base weight. The design effect for these cases was approximated as 1+cv2.
The effective sample size (n1) was computed as the unweighted sample
size divided by the design effect. The effective sample size for the cell frame
dual service cases (n2) was computed in an analogous way. The compositing
factor for the landline frame dual service cases was computed as n1/(n1
+ n2). The compositing factor for the cellular frame dual service
cases was computed as n2/(n1 + n2).
The
survey sample was then balanced to population totals for the Asian-American
adult population. The sample was balanced to match national net Asian American
adult population parameters from the 2010 American Community Survey public use
microdata sample (ACS PUMS) for gender by age, gender by education, education by
age, region, ethnic group by nativity, ethnic group by gender by age, and
ethnic group by gender by education. The ethnic group categories were Chinese
alone, Filipino alone, Asian Indian alone, Japanese alone, Korean alone,
Vietnamese alone, and Other Asian, including those that identify with more than
one Asian group. In addition, the sample was balanced to Asian-American adult
telephone service estimates from an analysis of the January-June 2011 National
Health Interview Survey. The distribution of the calibrated weights was
examined for any extreme values. The distribution of the final weights was
truncated at the median + (6 x the interquartile range). This trimming was
performed in order to reduce extreme variance in the weights and ultimately
improve the precision of the weighted survey estimates. The sum of the final
weights was set to equal the total number of net Asian-American adults based on
the 2010 ACS PUMS.
Due
to the complex nature of the 2011 Asian-American Survey, formulas commonly used
in RDD surveys to estimate margins of error (standard errors) are
inappropriate. Such formulas would understate the true variability in the
estimates. To account for the complex design, a repeated sampling technique—specifically
jackknife delete two repeated replication, JK-2—was used to create replicate
weights for this study. The subsamples (replicates) were created using the same
sample design, but deleting a portion of the sample, and then weighting each
subsample up to the population total. A total of 100 replicates were created by
combining telephone numbers to reduce the computational effort. A statistical
software package designed for complex survey data, Stata v12, was used to
calculate all of the standard errors and test statistics in the survey.
Supplemental Tables
Footnotes:
50 Based on the 2010 American Community Survey. (return to text)
51 Experian is a commercial database company that uses an analysis of first and
last names to identify households likely to include a person or persons of a
given ethnicity. For more see the section on list stratum below. (return to text)
52 For more on the methodology of that survey see Pew Research Center for the
People & the Press. 2011. “Muslim
Americans: No Signs of Growth in Alienation or Support for Extremism.” (return to text)
Washington, D.C.: August.
Photo Credits from left to right: © Radius Images/Corbis, © Image Source/Corbis, Istockphoto and © 2010 Getty Images