Energy and Environment Project 2012-2013
Mongolia, 2012 - 2013
Get MicrodataIdentification
MNG_2012_MCC-EEP_v01_M
Energy and Environment Project 2012-2013
Stove Subsidies Component
| Name | Country code |
|---|---|
| Mongolia | MNG |
Independent Impact Evaluation
This impact evaluation assessed stove performance and impacts under real-world usage conditions. It was designed to answer the following questions:
Evaluation question 1: How do energy-efficient products impact ambient air pollution levels, and health and income of residents in Ulaanbaatar? Specifically:
a) How does the use of MCA stoves affect fuel usage and expenditures?
b) Does the use of MCA stoves affect available household income?
c) What is the impact of MCA stoves on emissions of CO and PM2.5?
d) What is the impact of MCA stoves on indoor concentrations of CO and PM2.5?
e) What would be the estimated change in health for Ulaanbaatar residents?
f) How do MCA stoves affect household expenditures related to respiratory health problems?
Evaluation question 2: How do different MCA stove models and different patterns of usage affect the level of impact on ambient air pollution, and the health and income of households with MCA stoves? Specifically:
a) Do different MCA stove model types impact fuel expenditures, income, and PM2.5 emissions, under typical usage behavior?
b) Do deviations from expected MCA stove usage patterns impact air pollution, health, and income of households with MCA stoves?
c) Did the MCA stove program result in differential impacts on men and women?
d) Does possession of additional energy efficiency products such as vestibules or additional ger insulation modify the impact of MCA stoves on ambient air pollution, health, and income?
Since this program was a market-based intervention, households chose whether to purchase an MCA stove. Because a randomized intervention assignment was not possible and the evaluation was implemented after the project had started, a quasi-experimental propensity score matching (PSM) design was used to adjust for differences between those who did and did not choose to purchase an MCA stove. Matching on propensity scores enabled construction of treatment and comparison groups that were balanced along the observed characteristics, thereby providing a counterfactual for the intervention.
Key findings include:
-
Participants in the EEP stove subsidy program had 65% lower emissions of PM2.5 and 16% lower CO emissions, both statistically significant, compared to traditional stoves under typical usage conditions.
-
The EEP stove subsidy program reduced ambient PM2.5 concentrations over UB attributable to heating stoves by an estimated 30%, with largest reductions in highly polluted areas that were more heavily targeted by the program.
-
Factors known to affect coal consumption differed systematically between Ulzii, Khas, and Dul stove users.
-
The EEP stove subsidy program achieved high demand for energy-efficient stoves and satisfaction among stove users; however, some stove limitations remained barriers to satisfaction.
-
The EEP stove subsidy program did not achieve significant reductions in daily coal consumption under typical usage conditions.
-
Very low compliance with MCA stove operation instructions may have contributed to lack of reduced coal consumption.
-
Significant reductions in coal use were observed when households used MCA stoves according to instructions.
-
Households using MCA stoves enjoyed significantly higher indoor temperatures, suggesting that users may be sacrificing fuel economy for comfort.
-
There is no evidence that the EEP stove subsidy program achieved reductions in overall coal expenditures.
-
MCA stove owners in gers with better insulation used less coal than traditional stove owners.
-
Observed emissions reductions may have contributed to substantially fewer cases of air pollution-related respiratory illness and related costs.
Sample survey data [ssd], air quality measurements
Households
Version
Anonymized dataset for public distribution
Scope
Coverage
The sample is restricted to residents of Ulaanbaatar's Bayangol, Bayanzurkh, Chingeltei, Khan-Uul, Songino Khairkhan, and Sukhbaatar Districts. The MCA program only targeted Ulaanbaatar residents in all but Bayangol District. The treatment group sample is a random sample of MCA stove owners selected from complete stove sales lists of Khan Bank and Xacbank. Comparison households (traditional stove owners) were randomly selected from the Ministry of Labor and Social Welfare's 2010-2011 Proxy Means Test (PMT) data, a census of all Ulaanbaatar ger area households that was designed to assess poverty levels.
Households in Ulaanbaatar Mongolia's Bayangol, Bayanzurkh, Chingeltei, Khan-Uul, Songino Khairkhan, and Sukhbaatar Districts. Primary stove tender was the main respondent.
Producers and sponsors
| Name |
|---|
| Social Impact |
| Name |
|---|
| Millennium Challenge Corporation |
Sampling
A sample size of 1096 was estimated to obtain 100% power to detect a 3.6 kg difference in daily coal use between traditional stove users and those using each of three different MCA models, accounting for expected attrition of households from data collection phase 1-phase 3. The treatment group sample is a random sample of MCA stove owners selected from complete stove sales lists of Khan Bank and Xacbank. Comparison households (traditional stove owners) were randomly selected from the Ministry of Labor and Social Welfare's 2010-2011 Proxy Means Test (PMT) data, a census of all Ulaanbaatar ger area households that was designed to assess poverty levels. The complete records from the Bank and PMT lists were stratified by dwelling type (ger or house) and stove type (Dul, Khas, Ulzii, or traditional), and households were randomly sampled within each stratum. While the sample was initially divided equally into dwelling and stove type strata, this allocation was adjusted during data collection since it was found that very few gers had Khas stoves, because the Khas stove was marketed as more suitable for houses due to its larger size. The Ger-Khas stratum was therefore reduced to only 15 observations. This allowed a reallocation of the survey sample to the other groups, increasing the size of the other strata to at least 150, thus increasing power.
At the start of Phase I data collection, a variety of challenges required that 35% of sampled households be replaced. First, there were numerous discrepancies in the household addresses within the bank and PMT lists on which the sampling framework was based. Of the households that data collectors attempted to interview, 13% identified through either list were actually living at another address, either because they moved or because an incorrect address was given or recorded in the lists; 4% of the listed addresses could not be located at all. In 2% of cases, enumerators discovered that the sampled home was actually an apartment building or business address, which were not eligible for the stove evaluation. While some of the households sampled from the PMT list would have moved since the time of the PMT survey two years prior to this evaluation, this would only explain a portion of discrepancies. Other reasons for sample replacements were address duplication in list (1%); no person home after 3 visits (3%); refusal to participate (3%); use of other improved stove model apart from MCA stove (4%); and household did not live in expected dwelling type per sampling list (5%).
Interviews were completed with 1,123 households in Phase I. There was a 6% attrition rate from Phase I to III, largely due to respondent fatigue, relocation, or continued absence of the intended respondent from the home. The final valid household survey sample at the end of the third phase of data collection was 1,057.
The expected sample size was 1096. At the start of Phase I data collection, a variety of challenges required that 35% (n=604) of sampled households be replaced. First, there were numerous discrepancies in the household addresses within the bank and PMT lists on which the sampling framework was based. Of the households that data collectors attempted to interview, 13% identified through either list were actually living at another address, either because they moved or because an incorrect address was given or recorded in the lists; 4% of the listed addresses could not be located at all. In 2% of cases, enumerators discovered that the sampled home was actually an apartment building or business address, which were not eligible for the stove evaluation. While some of the households sampled from the PMT list would have moved since the time of the PMT survey two years prior to this evaluation, this would only explain a portion of discrepancies. Other reasons for sample replacements were address duplication in list (1%); no person home after 3 visits (3%); refusal to participate (3%); use of other improved stove model apart from MCA stove (4%); and household did not live in expected dwelling type per sampling list (5%).
Following these replacements, 1,125 valid samples were obtained for Phase 1. Attrition dropped this number to 1079 in Phase 2 adn 1057 in Phase 3.
No weights were used.
Survey instrument
The HOUSEHOLD SURVEY was administered to the individual most responsible for tending the stove, although other household members could assist in answering demographic questions if necessary. After obtaining informed consent, enumerators gathered information on each dwelling's physical characteristics such as construction materials and insulation. Enumerators measured dimensions of the room in which the main stove was located to allow estimation of the heating space volume. Demographic characteristics were collected for the full household roster, including information on age, gender, education, marital status, as well as employment status and income in the prior month from work, pensions, and other allowances for every household member. A general household expenditures section was added to the survey during Phases II and III to further document potential economic impacts beyond changes in income. Though the survey methods and the sample size were not designed to measure health outcomes reliably, respondents were asked to report current respiratory, cardiac, and dermal symptoms experienced by household members in vulnerable age groups (<5 and >60 years old), to collect some data on symptoms that could be associated with air pollution.
The survey captured the ownership and use of up to three stoves within each home, as well as any other heating and cooking devices. In addition, data were gathered on stove usage for cooking and heating, any modifications made to MCA stoves, and criteria-based personal preferences between MCA and traditional stoves. To determine whether MCA stove owners were compliant with lighting instructions, respondents were asked an open-ended question about how they light their stove, and enumerators recorded pre-coded responses.
Respondents reported recent expenditures on truckloads of coal and wood since the last data collection visit (for Phase I data collection, since June). The types of coal and wood purchased were also reported. Enumerators showed photos of truck sizes to help the respondent estimate the quantity purchased. Total purchases of coal and wood by the sack were also reported for the previous week and the past two weeks. Types of coal and wood and the per-sack prices for the most recent purchase were also reported.
To estimate the quantity of fuel used daily and the number of fueling events, cold starts, and warm refuelings, enumerators asked the main stove tender to recall the times of each fueling event in the 24 hours preceding the interview. For each event, enumerators asked the respondent the time, purpose (heating, cooking, or both), whether the stove was still warm or had unburnt coal or embers present, and the quantities of each fuel type used. Any fuelings in which embers or unburnt coal were present were considered warm refuelings. To further verify fuel quantities, respondents were asked to put the amount of coal they used into a bag or bucket, which enumerators then weighed with handheld digital scales and recorded after subtracting the weight of the container.
Data collection
| Start | End | Cycle |
|---|---|---|
| 2012-10-20 | 2012-11-08 | Phase I |
| 2013-01-12 | 2013-01-26 | Phase II |
| 2013-03-01 | 2013-03-31 | Phase III |
| Name |
|---|
| joint venture between Robust LLC and the Institute of Philosophy, Sociology and Law |
| Mongolian University of Science and Technology (contracted by Social Impact) |
For the household survey, teams of 2 enumerators visited each sampled household. One was the primary interviewer while the other would focus on household structure measurements and weighing coal and providing support to the main interviewer. Both men and women played both roles.
Supervisors assigned households to enumerator teams, responded to impromptu questions from the field, and reviewed data quality. Respondents were also free to call supervisors (numbers were provided through the consent process) if they had questions about the study.
For the air quality data collection, two teams of three set up equipment in the homes. Equipment readings were downloaded in a laboratory at MUST with assistance from a team leader.
Household data were collected using pen and paper in three phases to capture changes in stove usage patterns throughout the winter of 2012-2013 to account for temperature fluctuations. Data collection was contracted by MCA to be conducted by a joint venture between Robust LLC and the Institute of Philosophy, Sociology and Law (JVRIPSL). The data collection was implemented by 15 teams of two enumerators each, managed by three supervisors. All enumerators completed a three-day training prior to Phase I that covered research ethics, data collection instruments, and protocols. One-day refresher training was provided before Phases II and III of data collection. Interviews lasted approximately 1 hour. Upon verification of contact information in Phase I, enumerators pre-arranged convenient appointment times with the panel of stove tenders during Phases II and III.
Household air quality data were collected by trained students at Mongolian University of Science and Technology who set up air quality sampling devices in a sampled home in the evening around 6pm. The sampling equipment took continuous measurements throughout the evening until the same team returned the following morning to retrieve the equipment. No personal data were collected during this time. Readings from the equipment were recorded on paper and internally by the devices.
Data processing
The data collection firm JVRIPSL was responsible for data cleaning. They performed skip logic checks. In addition, after each Phase, Social Impact sent data quality monitoring and logic checks findings to the JVRIPSL to highlight extreme outliers or skip logic errors that might indicate a data entry to data collection error. JVRIPSL reviewed paper forms and audio recordings of interviews to correct or rule out data entry errors, verified feasible outliers, and at times followed up with households again to obtain the accurate response.
Access policy
Millennium Challenge Corporation
Millennium Challenge Corporation
http://data.mcc.gov/evaluations/index.php/catalog/133
Cost: None
Data Access
Use of the dataset must be acknowledged using a citation which would include:
Greene, Leslie; Turner, Jay; Edwards, Rufus; Cutler, Nathan; Duthie, Michael; and Olga Rostapshova. Social Impact 2012-2013 household survey data in support of the impact evaluation of the MCA Mongolia Energy and Environment Project Energy-Efficient Stove Subsidy Program.
Contacts
| Name | |
|---|---|
| Monitoring & Evaluation Division of the Millennium Challenge Corporation | impact-eval@mcc.gov |
Metadata production
DDI_MNG_2012_MCC-EEP_v01_M
| Name | Affiliation | Role |
|---|---|---|
| Millennium Challenge Corporation | Review of Metadata | |
| Social Impact | Independent evaluator | Documentation of Study |
2014-09-14
Metadata version
Version 1.1 (Original 2014-09-14)
Version 2.0 (May 2015). Edited version based on Version 01 (DDI-MCC-MNG-EEPIE-2014-v01) that was done by Millennium Challenge Corporation.
Program participants were consumers residing in Mongolia's Bayanzurkh, Chingeltei, Khan-Uul, Songino Khairkhan, and Sukhbaatar Districts who purchased energy-efficient residential heating stoves.
Survey participants were the primary stove tenders of either the MCA energy-efficient stoves or of traditional stoves (comparison group).