The key distinction between impact evaluation and other monitoring and evaluation techniques is that impact evaluation seeks to isolate the causal relationship between interventions and the welfare or wellbeing of beneficiaries. Given the objective of MCC to enhance economic growth, wellbeing will generally be captured by household consumption, assets, or income, at the household level, but occasionally at the individual level with measures such as assets or income to investigate gender effects. Since there are many factors influencing households' consumption, income, and wellbeing in a given year, including multiple projects within an MCC Compact, often a simple before-and-after comparison can lead to a misleading or incorrect assessment of project impacts. The challenge of impact evaluation, therefore, is to identify suitable comparison groups to compare with beneficiaries. Randomization is considered the gold standard, since it is the best tool available to address confounding observable and unobservable factors in a research design by ensuring their balance across treatment and control groups. However, randomization was not feasible for the PAP evaluation: legally the individuals living on the land turned into irrigated land were automatically beneficiaries. Therefore, the evaluation team is using PSM to evaluate the total impact of the AIP on the PAPs versus individuals and households living in the same general geographic area who had similar characteristics to the PAP prior to the program but that did not directly benefit from getting land.
The use of PSM relies on the untestable assumption that there are no unobservable differences between villages with PAPs and other communities in the area of the ON. If unobserved factors influence the rate of adoption between the treatment and comparison groups, then this could bias the outcome indicators. This report provides a number of analyses to see if the estimates are robust to changes in how the analysis is done, and uses both baseline and follow up data to give us more confidence in the estimates. Nevertheless, it is important that readers keep in mind that we have less overall confidence in these impact estimates than if the project had been evaluated using the RCT method.
There are some aspects of the AIP which are not evaluated through rigorous impact evaluation, such as the Niono-Goma road and the improvements to the main water conveyance system. All residents living near the road will have benefited from the program, not just the PAPs. For the water conveyance system, there are many more beneficiaries of this component of the project (all farmers with irrigated land in the ON), and this impact evaluation will not, and was not designed to, provide any estimate of those project impacts.
Kind of Data
Sample survey data [ssd]
Unit of Analysis
Community, Household, Individual
Anonymized dataset for public distribution
There are two groups that benefited from the AIP: those who were already living on the land converted into irrigated land, or PAPs, and individuals who applied for and won land through a lottery, or New Settlers.
Producers and sponsors
Innovations for Poverty Action
Millennium Challenge Corporation
The stratified, two stage cluster sample which was chosen should provide sufficient variation in the comparison group of 115 villages to identify the project impacts in the 33 Alatona villages that will benefit from the AIP. This large number of villages, which represents an 18% sample from the 32 communes that we identified as part of the survey zone, is necessary to identify program impacts among the numerous interventions planned for the AIP using propensity score matching. Because propensity score matching requires careful identification of similar households in both beneficiary comparison groups, a large number of “candidate” households are necessary in the comparison group to insure that good matches can be made.
One of the estimators employed is the Epanechnikov kernel-matching estimator for the average treatment effect on the treated. The advantage of this estimator is that it gives relatively higher weight to "closer" matches and lower weight to matches that are less close in the calculation of the average treatment effect on the treated.
The study also uses entropy matching, an iterative process through which weights are constructed such that the weighted comparison mean is equal to the treatment mean for all covariates. By focusing on reweighting the data rather than the inclusion or exclusion of potentially important covariates, the technique potentially reduces bias from omitting key variables and also removes concerns about limiting a wide potential set of covariates due to imbalance across the distribution of propensity scores generated from un-weighted samples.
Dates of Data Collection
Follow Up 1
Follow Up 2
A total of 18 surveyors, four Team Leaders and one field manager and 1 back-checker was employed.
Data Collection Notes
The survey was carried out using a Computer Assisted Program Interview with the Blaise software. The PAPs follow-up survey collected data from 788 household in both treatment and control group. The survey lasted 26 days and was carried out by four teams.
Environment and Social Development Company
The questionnaire design links the objectives of the AIP with the evaluation strategy, which is essential to the production of a quality data set useful for the AIP evaluation. The survey instrument was designed as three distinct questionnaires: community, men and women. A similar survey instrument was used across all PAP surveys.
The community questionnaire collected demographic and physical characteristics of the community in addition to information about the functioning of markets (migration and agriculture), access to infrastructure, and the quality of the infrastructure (health and education) that exists. In the Agriculture module, community level information with respect to the functioning of farmers' cooperatives, access to agricultural inputs, and management of irrigation plots (collection of water fees, community level investment, land tenure and transactions) was collected.
In 2009, ESDCO did a single data entry of the questionnaires. Unfortunately, during the baseline data cleaning process, IPA discovered too many data entry errors, which led to the necessity of a double entry process. The double data entry was conducted by CAREF from May to October 2010. The same questionnaire was entered twice by two different agents. After the two entries, the data entry coordinator proceeded to comparisons between the two entries and printed a list of the discrepancies. Three others agents were in charge of going back to the questionnaire to check which of the two entries was correct and replace the wrong entry by the correct one. Then, the data entry coordinator proceeded to a final comparison in order to make sure that all discrepancies were removed.
Subsequent data collection was undertaken through a contract between MCA and IPA directly. IPA implemented the surveys using netbooks, thus eliminating the need for data entry. This also allowed the impact evaluation team to properly put into place data quality assurances.
Monitoring & Evaluation Division
Millennium Challenge Corporation
Innovations for Poverty Action. 2013. Alatona Irrigation Project Impact Evaluation.