Watershed Management and Agriculture Support Project 2018, Independent Performance Evaluation

A theory-based approach was employed to conduct the post-compact performance evaluation of the WMAS Project. This approach examines the entire project causal chain or logical framework, from inputs to outputs and outcomes to possible impacts, to explore how and whether inputs were correctly designed to lead to the expected goals. The theory-based approach was used to assess the effect of a program in a way that accounts for both the underlying causal mechanisms and implementation processes. The objective was to obtain information not only on what works and what does not, but also on how and why the project succeeded or failed based on rigorous academic evidence.

With respect to project design, the Evaluator assessed whether the project was adequately designed to achieve the expected outcomes. To this end, the Evaluator took a holistic approach with a focus on relevancy and the need/demand for the project activities. The Evaluator reviewed related literature to pinpoint any conceptual shortcoming in the causal chain e.g. institutional / organizational aspects which may have influenced project outcome. In this context, the ERR calculations were reviewed, underlying assumptions were examined to assess whether the assumptions remained valid during the Compact period or if unpredicted events happened that violated the ERR assumptions. The Evaluator looked at any perceived risks and any mitigation strategies put in place to manage those risks, as well as whether contextual factors were considered.

With respect to project implementation, the Evaluator evaluated the overall adherence to the design. It assessed whether the intervention was carried out as planned, and in the same way everywhere within one island and across islands. From this perspective, the Evaluator examined several factors, in particular the quality of the construction, procurement procedures, supervision, choice of regions, environmental conditions, and the role of different entities and their coordination in the implementation process, as well as follow-up procedures. The Evaluator also assessed the quality of the training program and technical assistance, the reasons for implementation delays, factors that influenced the performance of different project participants during project implementation, etc.

With respect to project outcomes, the Evaluator evaluated whether the expected results were achieved (i.e. effectiveness), and whether these results have been sustainable during the post compact period (i.e. sustainability). The collected data included a large volume of administrative data on all the project activities during the mission. Primary data was also collected through surveys of project participants, such as farmers and traders, to gain an informed insight into the outcomes. The evaluation assessed how the three activities under the WMAS project interacted with each other along with contextual factors to achieve the Compact's goal in order to capture a complete picture of the project's performance. The performance assessment was designed to consolidate the findings, while accounting for the links and interrelationships between each project activity. Moreover, lessons were drawn from the overall project evaluation.

The issue of attribution was addressed through contribution analysis. The attribution issue for the WMAS project was heightened by the lack of baseline data that would incorporate a valid counterfactual. 229 farmers in Santo Antão, 59 farmers in São Nicolau and 49 farmers in Fogo were, for instance, expected to gain increased access to water and credit to install drip irrigation. The contribution analysis addressed causal inference by looking for consistency of outcomes with the program theory while assessing/ruling out alternative explanations. The notion of contribution stems from the view that an intervention works alongside contextual factors to produce the observed outcomes. The attribution question can therefore be equivalent to asking what difference the program makes in bringing about the observed outcomes, and whether the program played any catalytic role that resulted in a specific outcome.

The performance evaluation followed a mixed methods approach, combining quantitative and qualitative data. The quantitative component consisted mainly of analysis of secondary data collected from national surveys, Performance Indicator Tables, available documents on the project activities, data provided by MCA Cape Verde and MCC, and survey data collected from farmers and traders. The qualitative component includes the literature review and the analysis of stakeholder interviews conducted during the fieldwork.

Evaluation Questions

The main evaluation questions for each activity are divided into two components as per the evaluation framework: Effectiveness and Sustainability.

The project's design aligns with academic literature and seems relevant to the government's priorities. However, the design did not fully consider the country's overall growth dynamics and agriculture suitability to bring optimal benefits. All infrastructure activities were concluded before the end of the Compact, but sequencing of the construction and quality control was inadequate in some cases. For instance, some reservoirs were made before the boreholes were in place and, therefore, could not provide water immediately to farmers who already had bought drip irrigation equipment. Because of leakages caused by poor construction, some of the reservoirs could not hold water for long. The training program under the agribusiness activity increased skills and knowledge of extension workers and farmers, but it was reported to be insufficient to have the expected impacts.Survey data shows that of the trained farmers, 22% reported that they would expect improvement in content, 64% reported that they would expect improvement in the duration of the training, and 16% reported the need for improving language issues. Similarly, the training program for participating MFIs did not include enough training at the operational level required for loan underwriting.