Performance Evaluation: Mozambique Farmer Income Support Project 2014

Trees

For the epidemic zone, the evaluation estimated the impact of FISP on disease prevalence and estimated the consequent impact on coconut production and farmer incomes. In addition, the evaluation also assessed the pre-conditions for future increase in income by examining whether the seedlings planted through FISP survived.

To assess these impacts, because the ideal approach of randomly assigning treatment to farmers was not feasible, the evaluation employed the next best approach. It identified geographic areas that were very similar at baseline and were otherwise split into treatment and comparison areas by some external “quasi-random” factor (the phytosanitary barrier), and matched a subset of these areas on disease prevalence and distance to coast. This approach is sometimes called a “border discontinuity” design in the applied microeconomics literature. As mentioned in Section 2.3, the phytosanitary barrier was a north-to-south imaginary line that FISP delineated, separating the areas with less than 10 percent disease prevalence (on the west side of the barrier) from the areas with more than 10 percent disease prevalence (on the east side). FISP worked on the west side of the barrier. Defined this way, the project worked in the entire epidemic zone.

However, in effect this barrier was imperfect because while it is true that generally (or on average) the disease prevalence is greater near the coast, the disease prevalence has a mosaic pattern. This means that even on the east side of the barrier, there can be areas with disease prevalence lower than 10 percent. Furthermore, the barrier was imperfectly drawn even along the perceived divide of 10 percent prevalence, as it was drawn along natural barriers such as water bodies and roads. These characteristics of the barrier and how the FISP treatment areas were defined give a unique opportunity to identify the counterfactual areas in the areas just east of the barrier. Accordingly, we used the 2008 TTI aerial photography data (TTI Production, 2009) to identify areas with similar disease prevalence as proximate as possible to either side of the barrier, with the areas on the west side serving as treatment areas, and areas on the east side serving as the comparison areas .

From among all FISP project areas on the east side of the barrier, and the non-project areas on the west side of the barrier, we identified areas where the disease prevalence was between 0 and 15 percent. These treatment and comparison areas amounted to 93 project and 65 non-project census enumeration areas (EAs) very close (within 73 kilometers) to the phytosanitary barrier that shared similar disease prevalence. Within these areas, we further refined the sample and identified approximately 24 nearest neighbor pairs of EAs matched on baseline disease prevalence and distance to coast. Our sample selection procedure, which resulted in 48 EAs, limits analysis and direct applicability to areas close to the phytosanitary barrier. The pairs were matched on:

Same district

Average baseline CLYD prevalence in 2008, in the 5-kilometer radius
Distance to coast.

No counterfactual EA was “matched” with more than one treatment EA. A small wrinkle in this identification strategy was that the FISP phytosanitary barrier was reevaluated a total of five times throughout the project, causing the barrier to shift during the course of the implementation (see Figure 2-1). To ensure that all comparison areas did not receive treatment, we have chosen comparison located east of the 2012 barrier, which is the furthest east the barrier was shifted over the course of the project.

It is important to note that in three out of the eight FISP districts there was no barrier drawn: in Pebane, Moma, and Angoche, the barrier was the coast, and FISP worked in these districts in a way that made it difficult to identify non-project areas. There was no barrier drawn in the district of Chinde, but epidemic and endemic zones (as well as project and non-project areas) were geographically separated in a way that allowed us to include the district in our sampling frame. As a result, our quantitative evaluation is focused on five out of the eight FISP districts (Nicoadala, Namacurra, Maganja da Costa, Chinde, and Inhassunge). In the remaining three districts in the Nampula district, we relied on a qualitative survey to draw lessons about the project's efficacy.

In sum, the phytosanitary barrier distinguished between epidemic FISP areas and epidemic non-FISP areas. We identified 93 FISP census EAs on the west side of the barrier and 65 non-project EAs on the east side of the phytosanitary barrier but very close (within 73 kilometers) with similar baseline disease prevalence (using the 2008 aerial photography data from TTI Production, 2009). From these, 24 pair-wised matched project and non-project areas serve as our treatment and comparison groups. Their baseline similarity, geographical similarity, and haphazard separation by the phytosanitary barrier create the ideal conditions for a quasi-experimental evaluation. This quasi-random approach identifies the impact because difference in outcomes can be attributed only to the intervention; other potentially confounding factors are similar across the phytosanitary barrier, due to geographic proximity and because of matching (Section 5.3 provides more details on the results of the matching).

Households

We designed two separate quantitative evaluations to assess the activities in the epidemic zones and the endemic zones.

Evaluation of Epidemic Zone Program: In the epidemic zones, the evaluation used a border discontinuity design. Specifically, it identified geographic areas that were very similar at baseline, except that they were split into treatment and comparison areas by an external “quasi-random” factor-the phytosanitary barrier. The phytosanitary barrier was a north-to-south imaginary line that FISP delineated, separating the areas with less than 10 percent disease prevalence (on the west side of the barrier) from the areas with more than 10 percent disease prevalence (on the east side). FISP did not conduct any epidemic zone activities east of this barrier. Despite the clear distinction of the barrier for implementation activities, the disease had a mosaic pattern, implying that the barrier was imperfect at delineating CLYD prevalence levels. This feature aided our evaluation, because we were able to select comparison areas to the east of the barrier that had roughly the same qualifications for selection (baseline disease prevalence) as the treatment areas. Among a subset of the census enumeration areas (EAs) on both sides of the barrier that had less than 15 percent disease prevalence, we matched EAs based on distance from coast (which also proxies for distance to barrier) and baseline disease prevalence. This strategy was feasible in five out of the eight FISP districts. In the remaining three districts in the Nampula province, where the epidemic and endemic zone programs were combined and there was no barrier separating project areas, we relied on qualitative data to assess the project.

To assess impact on outcomes related to household income coconut production and seedling survival, the unit of analysis was households. We sampled 800 households across 24 treatment and 24 comparison EAs. After accounting for nonresponse, missing observations, and outliers, we obtained a study sample of 666 usable observations. For each household we gathered information on their pre-intervention demographic characteristics and information to measure the key outcomes: seedling survival, coconut production, and income.

To assess the impact of FISP on disease prevalence, we surveyed 16,000 trees that were sampled from the same enumeration areas as the household survey in the epidemic zones. We first identified the land area within the matched EAs that had tree cover, based on the 2008 aerial photography data. Within these areas, we randomly sampled 400 grids, each 100 by 100 meters, and within each grid we drew a sample of 40 trees in four clusters. For each tree in the grid, the tree enumerators visually inspected whether the tree was healthy; had only CLYD (and not beetle infestation); had beetle infestation only; or had CLYD and beetle infestation. The enumerators assessed beetle infestation also because CLYD is closely associated with increase in beetle infestation since dead trees - dead because of CLYD - can become a breeding ground for beetles.

To answer the question on FISP's impact on disease spread, however, we could not apply this method because that requires time series information on disease spread before, during, and after FISP. We conducted the analysis on two pairs of treatment and comparison areas for which we could find satellite data to estimate the prevalence of healthy trees between 2008 and 2014.

Evaluation of the Endemic Zone Program: In the endemic zones, which were entirely east of the phytosanitary barrier, the evaluation used a non-experimental method for selecting untreated geographic areas that, at baseline, were very similar to the treated areas. We matched pairs of treatment and comparison enumeration areas on baseline disease prevalence and distance from coast. The results of our matching and comparison of their self-reported pre-intervention characteristics suggest that the two groups are comparable and did not require any re-weighting to “look similar.”

In the endemic zone, we surveyed 720 households in 80 EAs across all districts. Of the 720 households surveyed, 561 observations were usable. In addition to information gathered for the epidemic zone farmers, for endemic zone farmers we also gathered information on their adoption, production, and sale of crops promoted by FISP.

Other Questions: To answer other questions, including the assessment of the BDF activity, the R&D activity, and the adaptation and adherence of FISP implementation to project goals, we conducted a desktop review of the literature. In addition, for the BDF activities and assessment of FISP's adaptation and adherence to project goals, we conducted interviews with key stakeholders, and focus group discussions and interviews with farmers and grant recipients.