The World Bank Listening to LAC (L2L) Pilot 2011- 2012, Using Mobile Phones for High-Frequency Data Collection
Latin America & the Caribbean, 2011 - 2012
Get MicrodataIdentification
LAC_2011-2012_L2L_v01_M
The World Bank Listening to LAC (L2L) Pilot 2011- 2012
Using Mobile Phones for High-Frequency Data Collection
| Name | Country code |
|---|---|
| Latin America & the Caribbean | LAC |
Other Household Survey [hh/oth]
The rapid and massive dissemination of mobile phones in the developing world is creating new opportunities for the discipline of survey research. The World Bank is interested in leveraging mobile phone technology as a means of direct communication with poor households in the developing world in order to gather rapid feedback on the impact of economic crises and other events on the economy of such households.
The World Bank commissioned Gallup to conduct the Listening to LAC (L2L) pilot program, a research project aimed at testing the feasibility of mobile phone technology as a way of data collection for conducting quick turnaround, self-administered, longitudinal surveys among households in Peru and Honduras.
The project used face-to-face interviews as its benchmark, and included Short Message Service (SMS), Interactive Voice Response (IVR) and Computer Assisted Telephone Interviews (CATI) as test methods of data collection.
The pilot was designed in a way that allowed testing the response rates and the quality of data, while also providing information on the cost of collecting data using mobile phones. Researchers also evaluated if providing incentives affected panel attrition rates. The Honduras design was a test-retest design, which is closely related to the difference-in-difference methodology of experimental evaluation.
The random stratified multistage sampling technique was used to select a nationally representative sample of 1,500 households. During the initial face-to-face interviews, researchers gathered information on the socio-economic characteristics of households and recruited participants for follow-up research. Questions wording was the same in all modes of data collection.
In Honduras, after the initial face-to-face interviews, respondents were exposed to the remaining three methodologies according to a randomized scheme (three rotations, one methodology per week). Panelists in Honduras were surveyed for four and a half months, starting in February 2012.
In Peru, households were randomly assigned to a communication mode (SMS, IVR, CATI), which stayed constant for all rounds (waves) of the survey.
Sample survey data [ssd]
- Households
Version
v01
Scope
The scope of the study includes rigorous testing using experimental design of:
- reliability testing
- criterion validity testing
- attrition rate
- cost of data collection
- incentives to respondents
Coverage
Peru and Honduras - Includes the entire national territory, with the exception of neighborhoods where access of interviewers is extremely difficult, due to lack of transportation infrastructure or for situations that threaten the physical integrity of the interviewers and supervisors (i.e. extremely high crime rate, warfare, etc.)
All the households that exist in the neighborhoods of Honduras, as reported by the 2001 Census. Institutions such as military, religious or educational living quarters are not included in the universe.
Producers and sponsors
| Name | Affiliation |
|---|---|
| Amparo Ballivian | World Bank |
| Joao Pedro Azevedo | World Bank |
| Will Durbin | World Bank |
| Name | Role |
|---|---|
| Gallup | Survey implementation |
| Name |
|---|
| Spanish Trust Fund for Latin America and the Caribbean |
| Rapid Social Response trust fund |
| Environmentally and Socially Sustainable Development trust fund |
| The Brightstar Corporation |
Sampling
Honduras
Honduras did not have an income oversample because the poverty rate is 60 percent, so oversampling 20 percent above the poverty rate would include a large portion of the middle class, which are not the most vulnerable in times of crisis.
The Honduras panel was built on a nationally representative sample of 1,500 households. The sample was drawn by means of a random, stratified, multistage design. The pilot used Gallup World Poll sampling frame.
Census-defined municipalities were classified into five strata according to population size:
I. Municipalities with 500,000 to 999,000 inhabitants
II. Municipalities with 100,000 to 499,000 inhabitants
III. Municipalities with 50,000 to 99,000 inhabitants
IV. Municipalities with 10,000 and 49,000 inhabitants
V. Municipalities with less than 10,000 inhabitants
Interviews were then proportionally allocated to these five strata according to their share among the country's population.
-
The first stage of the design consisted of a random selection of Primary Sampling Units (PSU's) within each of the five strata previously defined.
-
In the second stage, in each PSU, one or more Secondary Sampling Units (SSU's) were then selected.
-
Once SSU's were selected, interviewers were sent to the field to proceed with the third stage of the sample design, which consisted of selecting households using a systematic "random route" procedure. Interviewers started from the previously selected "random origin" and walked around the block in clockwise direction, selecting every third household on their right hand side. They were also trained to handle vacant, nonresponsive, non-cooperative households, as well as other failed attempts, in a systematic manner.
Peru
The Peru panel was built on a nationally representative sample of 1,500 households. The sample was based on the sampling frame for the National Household Survey (ENAHO) conducted by the Peruvian National Statistics Office (INEI) every three months.
In Peru, the sample selection was guided by the following criteria: (i) the sample should be representative nationally, and in urban and rural areas, and (ii) households close to poverty line should be oversampled because policy decisions in time of crises need to be especially mindful of the poor and vulnerable. For the purposes of this project, "close to poverty line" was defined as 40 percent of consumption distribution that symmetrically band the national poverty line: 20 percent above and 20 percent below. In 27 percent of Peruvian households monthly per capita consumption was below the moderate poverty line in 2010 (ENAHO).Those households whose monthly per capita consumption falls between 7 and 47 percent of the national distribution were oversampled.
The L2L sample frame comprises all the panel conglomerados from the fourth trimester of ENAHO 2010, or 281 conglomerados.
Detailed information about the sampling procedure is available in "Listening to LAC: Using Mobile Phones for High Frequency Data Collection, Final Report" (p. 65-69) and "The World Bank Listening to LAC (L2L) Pilot Project Sample Design for Peru."
In Honduras, 41% of recruited households failed to answer the first round of follow-up surveys. The attrition rate from the initial face-to-face interview to the end of panel study was 50%.
In Peru, 67 percent of recruited households failed to answer the first round of follow-up surveys. Attrition slightly increased with each wave of the survey (between 1 and 3 percentage points per wave), reaching 75 percent in wave 6.
As part of the survey administration process Gallup implemented a number of mechanisms to maximize the response rate and panelist retention. The following strategies were applied to respondents who did not replay first time:
- The surveys were left open for responses for up to 2 weeks after the original transmission of the survey (from original call in the case of IVR and CATI).
- First reminder was sent within 72 hours of first attempt (SMS and IVR).
- Second reminder was sent within 144 hours of first attempt (SMS and IVR).
- Call backs were made within 72 and 144 hours of first attempt (CATI); or
- Up to 2 call backs were made per appointment with respondent (CATI).
Also, in order to minimize non-response, three types of incentives were given. First, households that did not own a mobile phone were provided one for free. Approximately 127 phones were donated in Honduras, and approximately 200 phones were donated in Peru. Second, all communications between the interviewers and the households were free to the respondents. Finally, households were randomly assigned to one of three incentive levels: one-third of households received US$1 in free airtime for each questionnaire they answered, one-third received US$5 in free airtime, and one-third received no financial incentive (the control group).
Given the socio-economic focus of the L2L survey and the stark socio-economic differences that exist in Honduras, ensuring proportionality of socio-economic variables was in order. Age and educational attainment of the head of the household are typically used as proxy variables to a household’s socio-economic status due to their high correlation with income and other economic indicators. In the Honduras L2L survey, weighting was performed to correct for observed disproportions in these variables, relative to Census data.
Survey instrument
The following survey instruments were used in the project:
- Initial face-to-face questionnaire
In Peru, the starting point was the ENAHO (National Household Survey) questionnaire. Step-wise regressions were done to select the set of questions that best predicted consumption. For the purposes of robustness, the regressions were also done with questions that best predicted income, which yielded the same results. A similar procedure was done in Honduras, using the latest household survey deployed by the Honduran Statistics Institute, except that only best predictors of income were chosen, because Honduras did not have a recent consumption aggregate.
The survey gathered information on households' demographics, household infrastructure, employment, remittances, income, accidents, food security, self-perceptions on poverty, Internet access and cellphones use.
- Monthly questionnaires (SMS, IVR, CATI)
The questionnaires were worded exactly the same way, regardless of the mode, which meant short questions, since SMS is limited to 160 characters. A maximum of 10 questions had to be chosen for the monthly questionnaire. In addition, two questions sought to ensure the validity of the responses by testing if the respondent was a member of the household. Most questions were time-variant and each questionnaire was repeated to observe if answers changed over time. All questions related to variables that strongly affect household welfare and are likely to change in times of crisis.
- Final face-to-face questionnaire
Gallup conducted face-to-face closing surveys among 700 panelists. The researchers asked about issues the respondets had with mobile phones and coverage during the test. Panelists were also asked what would motivate them to keep on participating in a project like this in the future.
The questionnaires were worded exactly the same way, regardless of the mode, which meant short questions, since SMS is limited to 160 characters, unlike IVR and CATI.
Data collection
| Start | End | Cycle |
|---|---|---|
| 2011-10 | 2012-06 | Peru |
| 2012-02 | 2012-06 | Honduras |
| Name |
|---|
| CID-Gallup |
On average, 20 interviewers and 5 supervisors were trained in each country.
Gallup conducted a total of 12 training sessions with all its operational partners prior to launching the project. The training included:
- Survey methodology workshop aimed at identifying and reflecting on the most common sources of survey errors typically encountered in projects like the L2L
- Review of the L2L project’s objectives and methodology
- Panel recruitment and management workshop, including incentive administration and other attrition containment strategies
- Detailed review of (and hands on training on) the L2L face-to-face questionnaire and recruitment protocol
- Live SMS survey self-administration session
- Live IVR survey self-administration session
- Pilot face-to-face survey administration followed by a simulated recruitment for the panel among a group of actual households.
Honduras
One part of Honduran panel (n=600 households) was exposed to the three survey methods studied. They were also surveyed on a more frequent basis and for a relatively shorter period of time (four and a half months vs. six months in Peru). The surveys were administered in three time blocks, as follows:
-
Time One: This first administration took place between weeks 1 and 6 of the panel study. As part of "Time One," each panelist was surveyed three times. Each time was with the same exact questionnaire, but using a different survey method (SMS, CATI or IVR).
-
Time Two: This administration took place between weeks 12 and 16 of the panel study. As part of the "Time Two" block, each panelist was surveyed three times. Each time was with the same exact questionnaire used in Time One, and with the same three methodologies presented in the same order as in Time One.
-
In-Between: During the period between Time One and Time Two (weeks 7 through 11), panelists received four SMS surveys with questionnaires different than the ones used in "Time One" and "Time Two". However, for the purpose of the attrition analysis, only "Time One" and "Time Two" administrations are being considered.
The other part of the Honduran panel (n=900) was only interviewed by SMS after the initial face-to-face interview.
At each selected household, interviewers asked to speak to the person recognized by the family as the head of the household. If the head of the households could not be located after two attempts, interviewers proceeded to interview respondents 15 years and older who were permanent residents of the household.
Despite extensive planning by the World Bank and Gallup teams, no SMS messages went out during the week of April 23 due to an error by the SMS aggregator, which simply forgot to program the system to send out messages that week. However, in this case, the messages went out the next week and the response rates did not suffer as a result.
Peru
In Peru, households were randomly assigned to a communication mode (SMS, IVR, CATI), which stayed constant for all six waves of the survey.
Through a competitive bidding process, Gallup won the contract to implement L2L on the ground. Working with affiliates in Peru and Honduras, Gallup successfully completed the initial face-to-face interviews, monthly or bi-monthly surveys using mobile phones, and a final face-to-face interview of a sub-group of the initial sample. Gallup completed the face-to-face interviews as well as the IVR and CATI interviews in-house, and subcontracted the SMS interviews to an SMS aggregator.
Despite extensive planning by the World Bank and Gallup teams, no SMS messages went out during the week of April 23 due to an error by the SMS aggregator, which simply forgot to program the system to send out messages that week. However, in this case, the messages went out the next week and the response rates did not suffer as a result.
Data Access
Use of the dataset must be acknowledged using a citation which would include:
- the Identification of the Primary Investigator
- the title of the survey (including country, acronym and year of implementation)
- the survey reference number
- the source and date of download
Example:
Ballivian, A, Azevedo, J P, and Durbin, W. Latin America & the Caribbean - World Bank Listening to LAC (L2L) Pilot 2011- 2012 - Using Mobile Phones for High-Frequency Data Collection. Ref. LAC_2011-2012_L2L_v01_M. Dataset downloaded from [url] on [date].
Disclaimer and copyrights
The user of the data acknowledges that the original collector of the data, the authorized distributor of the data, and the relevant funding agency bear no responsibility for use of the data or for interpretations or inferences based upon such uses.
Contacts
| Name | Affiliation | |
|---|---|---|
| Amparo Ballivian | World Bank | aballivian@worldbank.org |
| Joao Pedro Azevedo | World Bank | jazevedo@worldbank.org |
Metadata production
DDI_LAC_2011-2012_L2L_v01_M_WB
| Name | Affiliation | Role |
|---|---|---|
| Development Data Group | World Bank | Survey documentation |
2020-04-03
Metadata version
v01 (April 2020)