Rural Water and Sanitation Supply 2010-2013

The impact evaluation design of the rural water intervention proposes two separate approaches so as to ensure a “defensive evaluation” that is, so that results can be obtained in spite of unfore­seen events or mishaps. One is a phased randomized control trial (RCT) in which 100 water sys­tems were allocated (roughly) equally to treatment (i.e., early treatment) and control (i.e., late treat­­ment) groups within each of the ten districts of Leso­tho. The other is a continuous treat­ment-variable (CTV) model in which the intensity of treatment can be estimated without the need to isolate a separate control group. As we shall see, while our revised design provides tech­niques to overcome the threats to internal validity that delayed implementation created for the RCT design, there are compelling reasons to give a greater role to the CTV approach as well as to carry out end­line surveys.

The original evaluation design, developed under NORC's first contract with MCC, focused on a randomized design, under which NORC, with MCA planned for a 6-9 month gap between the end of construction and rehabilitation of treat­ment water projects (in 50 Phase-A villages) and the start of water pro­jects in the control areas (in 50 Phase-C villages) to ensure sufficient time for impact to be realized in all outcomes of interest. A nine month gap would allow us to mea­sure long-term outcomes such as changes in productivity and income, while a six-month gap would limit the impact analysis to short-term and intermediate outcomes. Under the original design, all Phase-A villages had a largely similar construction timeline with concurrent start and end dates of construction; thus, it was reasonable to expect that there would be a nine-month (or, at a minimum, a six-month) lag between the end of construction of the 50 Phase-A villages and the start of construction of the 50 Phase-C villages.

Delays in the construction of Phase-A (early treatment) water systems, resulted in 11 treatment villages (Phase-A1 villages) under­going construction concurrently with the Phase-C control villages. This over­lap has called into question the validity of the original evaluation design and the effi­cacy of admin­istering an end­line. As Table 1 in the Evaluation Design Report demonstrates, construction was completed in only 70 per­cent of Phase-A vil­lages (34 of the 50) 9 months before construction commenced in Phase-C con­trol villages. For these 34 villages - which belong to the Phase-Arev group - the time of exposure to treatment before controls began receiving treatment between in January 2013 range from 10 to 20 months. Since the mid­line data collection preceded the start of construction in Phase-C villages (i.e., Novem­ber to December, 2012), duration of exposure to treatment by the time of midline data collection for these 34 Phase-Arev villages was about 9-19 months. Five villages in Phase Arev and all 11 Phase-A1 villages had not been exposed to treatment by the time of midline data col­lec­tion. The reduction of the treatment sample at midline by 30 percent has implications for the sta­tistical power of the evaluation design to detect impacts at the level gen­erally desired by MCC.

The Evaluation Design Report discusses the implications of construction delays for the evaluation design, indicates what evaluation questions can and cannot be addressed without an endline, and discusses the pros and cons of alternative design stra­te­gies to deal with the unforeseen complications in implementation.