The key distinction between impact evaluation and other monitoring and evaluation techniques is that impact evaluation seeks to isolate the causal relationship between interventions and the welfare or wellbeing of beneficiaries. Given the objective of MCC to enhance economic growth, wellbeing will generally be captured by household consumption, assets, or income, at the household level, but occasionally at the individual level with measures such as assets or income to investigate gender effects. Since there are many factors influencing households' consumption, income, and wellbeing in a given year, including multiple projects within an MCC Compact, often a simple before-and-after comparison can lead to a misleading or incorrect assessment of project impacts. The challenge of impact evaluation, therefore, is to identify suitable comparison groups to compare with beneficiaries. Randomization is considered the gold standard, since it is the best tool available to address confounding observable and unobservable factors in a research design by ensuring their balance across treatment and control groups. However, randomization was not feasible for the PAP evaluation: legally the individuals living on the land turned into irrigated land were automatically beneficiaries. Therefore, the evaluation team is using PSM to evaluate the total impact of the AIP on the PAPs versus individuals and households living in the same general geographic area who had similar characteristics to the PAP prior to the program but that did not directly benefit from getting land.
The use of PSM relies on the untestable assumption that there are no unobservable differences between villages with PAPs and other communities in the area of the ON. If unobserved factors influence the rate of adoption between the treatment and comparison groups, then this could bias the outcome indicators. This report provides a number of analyses to see if the estimates are robust to changes in how the analysis is done, and uses both baseline and follow up data to give us more confidence in the estimates. Nevertheless, it is important that readers keep in mind that we have less overall confidence in these impact estimates than if the project had been evaluated using the RCT method.
There are some aspects of the AIP which are not evaluated through rigorous impact evaluation, such as the Niono-Goma road and the improvements to the main water conveyance system. All residents living near the road will have benefited from the program, not just the PAPs. For the water conveyance system, there are many more beneficiaries of this component of the project (all farmers with irrigated land in the ON), and this impact evaluation will not, and was not designed to, provide any estimate of those project impacts.
Kind of Data
Sample survey data [ssd]
Unit of Analysis
Community, Household, Individual
Anonymized dataset for public distribution
Unit of Analysis
Community, Household, Individual
There are two groups that benefited from the AIP: those who were already living on the land converted into irrigated land, or PAPs, and individuals who applied for and won land through a lottery, or New Settlers.
Producers and sponsors
Authoring entity/Primary investigators
Innovations for Poverty Action
Millennium Challenge Corporation
The stratified, two stage cluster sample which was chosen should provide sufficient variation in the comparison group of 115 villages to identify the project impacts in the 33 Alatona villages that will benefit from the AIP. This large number of villages, which represents an 18% sample from the 32 communes that we identified as part of the survey zone, is necessary to identify program impacts among the numerous interventions planned for the AIP using propensity score matching. Because propensity score matching requires careful identification of similar households in both beneficiary comparison groups, a large number of “candidate” households are necessary in the comparison group to insure that good matches can be made.
One of the estimators employed is the Epanechnikov kernel-matching estimator for the average treatment effect on the treated. The advantage of this estimator is that it gives relatively higher weight to "closer" matches and lower weight to matches that are less close in the calculation of the average treatment effect on the treated.
The study also uses entropy matching, an iterative process through which weights are constructed such that the weighted comparison mean is equal to the treatment mean for all covariates. By focusing on reweighting the data rather than the inclusion or exclusion of potentially important covariates, the technique potentially reduces bias from omitting key variables and also removes concerns about limiting a wide potential set of covariates due to imbalance across the distribution of propensity scores generated from un-weighted samples.
Dates of Data Collection (YYYY/MM/DD)
Follow Up 1
Follow Up 2
A total of 18 surveyors, four Team Leaders and one field manager and 1 back-checker was employed.
Type of Research Instrument
The questionnaire design links the objectives of the AIP with the evaluation strategy, which is essential to the production of a quality data set useful for the AIP evaluation. The survey instrument was designed as three distinct questionnaires: community, men and women. A similar survey instrument was used across all PAP surveys.
The community questionnaire collected demographic and physical characteristics of the community in addition to information about the functioning of markets (migration and agriculture), access to infrastructure, and the quality of the infrastructure (health and education) that exists. In the Agriculture module, community level information with respect to the functioning of farmers' cooperatives, access to agricultural inputs, and management of irrigation plots (collection of water fees, community level investment, land tenure and transactions) was collected.
Environment and Social Development Company
Innovations for Poverty Action. 2013. Alatona Irrigation Project Impact Evaluation.