DDI_LSO_2010_MCC-RWSS_v01_M Millennium Challenge Corporation 2015-02-24 NADA Version 1.0 (February 2015) Version 2.0 (April 2015). Edited version based on Version 01 (DDI-MCC-LES-IE-WASH-2010-v01) that was done by Millennium Challenge Corporation. Rural Water and Sanitation Supply 2010-2013 MCC-RWSS 2010-13 LSO_2010_MCC-RWSS_v01_M NORC at the University of Chicago NADA Millennium Challenge Corporation Monitoring & Evaluation Division Independent Impact Evaluation Lesotho Rural water Impact evaluation Water Sanitation Randomization RCT The impact evaluation design of the rural water intervention proposes two separate approaches so as to ensure a “defensive evaluation” that is, so that results can be obtained in spite of unfore­seen events or mishaps. One is a phased randomized control trial (RCT) in which 100 water sys­tems were allocated (roughly) equally to treatment (i.e., early treatment) and control (i.e., late treat­­ment) groups within each of the ten districts of Leso­tho. The other is a continuous treat­ment-variable (CTV) model in which the intensity of treatment can be estimated without the need to isolate a separate control group. As we shall see, while our revised design provides tech­niques to overcome the threats to internal validity that delayed implementation created for the RCT design, there are compelling reasons to give a greater role to the CTV approach as well as to carry out end­line surveys. The original evaluation design, developed under NORC's first contract with MCC, focused on a randomized design, under which NORC, with MCA planned for a 6-9 month gap between the end of construction and rehabilitation of treat­ment water projects (in 50 Phase-A villages) and the start of water pro­jects in the control areas (in 50 Phase-C villages) to ensure sufficient time for impact to be realized in all outcomes of interest. A nine month gap would allow us to mea­sure long-term outcomes such as changes in productivity and income, while a six-month gap would limit the impact analysis to short-term and intermediate outcomes. Under the original design, all Phase-A villages had a largely similar construction timeline with concurrent start and end dates of construction; thus, it was reasonable to expect that there would be a nine-month (or, at a minimum, a six-month) lag between the end of construction of the 50 Phase-A villages and the start of construction of the 50 Phase-C villages. Delays in the construction of Phase-A (early treatment) water systems, resulted in 11 treatment villages (Phase-A1 villages) under­going construction concurrently with the Phase-C control villages. This over­lap has called into question the validity of the original evaluation design and the effi­cacy of admin­istering an end­line. As Table 1 in the Evaluation Design Report demonstrates, construction was completed in only 70 per­cent of Phase-A vil­lages (34 of the 50) 9 months before construction commenced in Phase-C con­trol villages. For these 34 villages - which belong to the Phase-Arev group - the time of exposure to treatment before controls began receiving treatment between in January 2013 range from 10 to 20 months. Since the mid­line data collection preceded the start of construction in Phase-C villages (i.e., Novem­ber to December, 2012), duration of exposure to treatment by the time of midline data collection for these 34 Phase-Arev villages was about 9-19 months. Five villages in Phase Arev and all 11 Phase-A1 villages had not been exposed to treatment by the time of midline data col­lec­tion. The reduction of the treatment sample at midline by 30 percent has implications for the sta­tistical power of the evaluation design to detect impacts at the level gen­erally desired by MCC. The Evaluation Design Report discusses the implications of construction delays for the evaluation design, indicates what evaluation questions can and cannot be addressed without an endline, and discusses the pros and cons of alternative design stra­te­gies to deal with the unforeseen complications in implementation. Lesotho The project was implemented in all 10 districts of Lesotho. On average, 25 villages within each of these districts were selected for construction of new rural water systems and accompanying hygiene and training activities. Individual, household Sample survey data [ssd] Bureau of Labor Statistics The sampling frame for the IEMS consists of all villages in Lesotho based on publicly available geospatial data and 2006 Census data. Information on administrative location, geo-coordinates, rural-versus urban designation and population was merged with publicly available physiographic and geographic data to be used as covariates in the sampling. From this central dataset, indi­vi­dual sample frames were designed and PSUs were selected for two central project components: rural water and urban/peri-urban water. For rural water, villages were the primary sampling units. The centralized frame dataset consists of the following variables: · Primary Sampling Unit identifying information: Village name, Village ID (both GIS ID and Census ID), Village Geo-coordinates (X and Y), Enumeration Area ID, Community Council, District, Constituency · General covariates: Population, average annual temperature, precipitation, vegetation produc­tivity potential, number of households, urban/rural designation Sampling Design The random sample selection was sequentially sampled without replacement in the form of a multistage cluster design as follows: There are two stages to the design for the rural water intervention. They cover the designation and selection of villages (PSUs, clus­ters) and households (SSUs): · Village sample. As described in the Evaluation Design Report, of the 250 villages in 10 districts selected by DRWS for the MCA rural water interventions, 100 villages (10 per district) were deemed "ready" for the intervention in 2008. Fifty of these 100 villages were randomly assigned to treatment (Phase-A), while the remaining 50 were assigned to the control group (Phase-C). Final implementation lists, however, only consisted of a random sample of 50 treatment villages, 48 control vil­lages. The village locations are shown on the map in Annex B in the Evaluation Design Report. The only difference between the treatment and control villages was that the control villages would receive treatment after a delay during the evaluation period according to the pipeline design. Unfortunately, infrastructure construction ran into delays in villages selec­ted for treatment in three districts (Mohkohtlong, Qacha's Nek, and Thaba Tseka). These districts are the most remote (and mountainous), hence, likely to be systema­tically different from the other treated vil­lages. The emergence of this group receiving later treat­ment required our evaluation design to be retrospectively divided into two treatment groups, T1 (earlier treatment) and T2 (later treat­ment). · Household sample. Within each treatment and control village a systematic ran­dom sam­ple of 13 households was selected. The interview was conducted with the head of the household or the person in the house­hold most knowledgeable about household water and sanitation issues. The main data sources for this evaluation are the Impact Evaluation Multipurpose Survey (IEMS) and the suite of Activity Monitoring Plan (AMP) surveys. The IEMS is a longitudinal analytic survey specifically designed to collect data for the impact evaluations of the Lesotho Compact health and water activities. Three rounds of the IEMS were originally proposed in the evaluation design: a baseline, midline, and endline. These three rounds were approved by MCC. To date, the Bureau of Labor Statistics (BoS), under an Implementing Entity Agreement (IEA) with MCA-Lesotho, has conducted the baseline and midline IEMS. The AMP surveys are a suite of checklists and data collection forms developed by Cowater, in conjunction with DRWS and MCA, to monitor the construction and training activities associated with the Lesotho WSP. Of these, NORC has refined and expanded three surveys to collect data needed for the impact evaluation. These surveys would be adminis­tered to community (household) members, VWHC members and Water Minders in rural water villages along with the endline IEMS. The sample for these surveys would overlap with the vil­lages and households in the IEMS. The purpose of the surveys, which the evaluator proposes to implement with the endline IEMS, are described below: · Hygiene Awareness & Latrine and Environmental Hygiene (HALEH). Assesses the house­hold's knowledge and practices of proper hygiene and sanitation. The questionnaire consists of two parts: PHAST awareness and observation of the VIP latrine and the household's envi­ronmental surroundings. The HALEH sample will consist of 13 interviews per village, admin­istered to house­­­holds that were interviewed during the IEMS baseline and midline. For efficiency HALEH will become an additional module of the IEMS question­naire at endline. The HALEH module will provide an endline assessment on the link between the level of hygiene training received by the beneficiary and hygiene and sanitation know­ledge and practices. · Village Water and Health Committee Functionality (VWHC). Assesses the degree to which the members of the VWHC are able to pro­vide monthly rou­tine operational man­age­ment to the village's water system. It also asses­ses the current sta­tus of the system's oper­a­tion. The VWHC ques­tionnaire will be administered to the committee in a group setting at endline. · Water Minder Expertise (WM). Assesses the level of training that the WM received by determining his level of knowledge in after care main­tenance and in repairs of the sys­tem. It will be administered to each Water Minder in the village (approximately 2-3) at endline. Millennium Challenge Corporation Millennium Challenge Corporation http://data.mcc.gov/evaluations/index.php/catalog/137 Cost: None