Water Management and Services 2006-2013

The agency contracted to carry out the surveys consistently failed to collect key data properly and the consultant hired to assess data quality did not detect this failure, leading to a re-worked evaluation design and sampling plan. This was developed using a multi-site hierarchical design to meet generally accepted levels of rigor: a statistical power of 80 percent at a level of significance of 5 percent, assuming that site and farmergroup effects together capture 70 percent of impact variance. Under a fixed-effects design, a minimum detectable effect size (MDES) of 20 percent would be achievable (US$1,000 spread over a 5-year period, or about $200 per year). Here, just 5 treated and 5 (synthesized) comparison farmers (SSUs) per PSU would be needed for a total sample size of 210. Under a randomeffects design (with a treatment effect variance of 5 percent), a MDES of 22-25 percent should be achievable. Here, an equal number of 8-10 treated and (synthesized) comparison farmers would be required for a total sample size of 336 to 420. These power calculations indicate that while overall impacts could be inferred for a sample pooled across the three watersheds, the only detectable disaggregated impact for this MDES range would be the overall impact on Santo Antão. Analogous power calculations find that gender-specific impacts could only be detected in a fixed-effects specification at an MDES of 24 percent and if the sample were pooled across the three watersheds. These results are due to the limited size of the comparison group and relatively lower number of female heads of household in the sample.