Green Prosperity: Cocoa Grant Portfolio 2017-2019, Independent Performance Evaluation
Indonesia, 2017 - 2019
Get MicrodataIdentification
IDN_2017-2019_MCC-GPCGP_v01_M
Green Prosperity: Cocoa Grant Portfolio 2017-2019
Independent Performance Evaluation
| Name | Country code |
|---|---|
| Indonesia | IDN |
The purpose of the performance evaluation is to understand the degree to which the Window 1 grants under the Green Prosperity Cocoa portfolio are meeting the objectives of the portfolio and to generate learning surrounding the implementation of these grants to date.The performance evaluation was done in Phase 1 before the grants concluded, and in Phase 2 one year after grant completion.
Evaluation Questions for Phase 1 were:
- To what extent have the GP Cocoa grants' (Cocoa Revolution, SCPP and EQSI) training approaches proven successful in improving farmers' knowledge, attitudes and practice of GAP/GEP?
- How has each grant progressed in achieving its short and medium-term outcomes (phase 1) and long-term outcomes (phase 2)?
- What evidence is there that results or outcomes of the GP Cocoa grants will be further scaled and sustainable, and what results appear to be less sustainable? Why?
- What aspects of the GP Cocoa grant approaches have proven to be most relevant in meeting the needs of the Indonesian cocoa sector?
Under Phase 2, the Evaluation questions were as follows:
- To what extent were the Theories of Change valid in achieving the overall project objectives?
- To what extent have the GP cocoa grants' (Cocoa Revolution, GP-SCPP, and EQSI) approaches and activities proven successful in improving farmers' knowledge, attitudes, and practice of GAP/GEP?
- How did the GP cocoa grantees monitor grant progress towards results and outcomes during implementation, and how did they use this information to manage project performance?
- What results or outcomes of the GP cocoa grants are likely to be sustainable and scalable, and what results do not appear to be sustainable and scalable?
The Phase 2 evaluation found:
- Successful implementation of grantee TOC was highly correlated with the presence of pre-existing supply chain interventions and technical assistance delivery infrastructure due to time and logistics constraints, as well as grantee focus.
- “Training” reinforced existing knowledge but grantees stressed the need for “coaching” to address intermittent problems. Efforts to mobilize credit for increased investment were largely ignored by farmers and weak price transparency limited improvements in market access. However, grantees noted bean quality improvements over time.
- Pre-existing knowledge management systems overcame initial challenges to guide program decision-making. New systems faced time and logistical constraints. For all grants, limited partner and GOI access resulted in some frustration, and post-project, each partner retreated to proprietary data control and competition.
- Sustainability certification and nurseries responded to market and producer demands, respectively. Fermentation for bean quality proved to be unviable due to logistical constraints and low-price incentives. Fostering smallholder capital investments for minimal productivity growth continues to be a key challenge to sustainability in the sector.
Sample survey data [ssd]
individuals, other (farming associations), community, government
Version
Anonymized dataset for public distribution
Scope
| Topic | Vocabulary |
|---|---|
| Agriculture and Irrigation | MCC Sector |
Coverage
In Phase 1, the evaluation team selected the provinces of Southeast Sulawesi, South Sulawesi, West Sulawesi and West Sumatra as target provinces due to the high numbers of cocoa farming households and participating farmers and also to cover the main GP-SCPP implementation "clusters." GP-SCPP implementation clusters are strategies tied to particular locations (provinces) where the private sector partners working with GP-SCPP in that location sign up to a particular approach and agreed to work together.
In Phase 2, the evaluation team replaced field visits to West Sumatra with visits to East Nusa Tenggara, another site where GP-SCPP was implemented. Selection of districts for data collection within each province was purposive, aimed at representing the major regions of the national cocoa production areas and all the grants involved in the Cocoa portfolio. The Cocoa Revolution program is only implemented in two districts, both of which were included in both Phase 1 and 2 evaluations. The EQSI program works in three districts, two of which are included in the evaluation. Four out of ten GP-SCPP districts in Sulawesi were selected, including two that overlap with the other projects, and 2 districts in East Nusa Tenggara were selected. Because of its larger scope and large number of project partners and clusters, more fieldwork was conducted at the GP-SCPP site.
In Phase1, the study population included farmers, buying station (collectors) and input suppliers (i.e. nurseries), farmer groups and associations, MCC/MCA-I project staff, grantee staff, field staff, goverment representatives, project management and private sector partners in Indonesia.
In Phase 2 the study population included the same beneficiary groups but since the grants had concluded it did not include grantee field staff or MCA-I.
Producers and sponsors
| Name |
|---|
| Social Impact |
| Name |
|---|
| Millennium Challenge Corporation |
Sampling
Both Phase 1 and 2 evaluations included the two districts where the Cocoa Revolution program was implemented and two out of the three districts where the EQSI program was implemented. Both Phse 1 and 2 included four out of 10 GP-SCPP districts in Sulawesi including two which overlap with the other project, though Phase 1 included 2 additional districts in West Sumatra, while Phase 2 included 2 additional districts in East Nusa Tenggara. Because of its larger scope and large number of project partners and clusters, more fieldwork is to be conducted at the GP-SCPP sites.
The individuals selected for key informant interviews (KII) includes those working in key positions for project stakeholders including project staff (management and technical staff), government (Bappeda and Department of Agriculture representatives), private sector partners, buyers/collectors, input suppliers (nurseries) and local community leaders/village heads. They were selected purposively based on comprehensive contact lists and input received from grantees and from MCC regarding appropriateness and level of project involvement. Project reports were also used to identify the key actors.
For beneficiary FGDs and the mini-survey, in each district two farmer groups were randomly selected from a complete list of all farmer groups in each selected district. Each group selected represented a different sub district and project field staff assisted the ET with contacting each farmer for recruitment for the FGD. The sample unit for the FGDs and the mini-survey was the farmer household represented by the farmer. Observation sites were selected using snowball sampling. Direct observations of buying stations were based on stations identified by private sector partners and community members in each region, and nurseries were selected based on community member and community leader feedback.
In Phase 1, a total of 16 FGDs, 148 mini-surveys, 20 direct observations and 65 KIIs were conducted. There was a total of 72.5% male respondents and 27.5% female respondents. There were eight FGDs done for GP-SCPP, four for CR, and four for EQSI; there was an average of 10.5 participants in each group, and an overall participation of 41.2% women and 58.8% men out of 165 total respondents. For the mini-survey, a total of 148 respondents participated, with 87 for GP-SCPP, 19 for CR and 42 for EQSI and an overall participation of 38% women and 61.4% men.
In Phase 2, the evaluation team conducted 55 KIIs (82% male/18% female), 18 FGDs, 25 direct observations (84% male/16% female) and 115 mini-surveys. There was a total of 70% male respondents and 30% female respondents. There were nine FGDs for GP-SCPP, six for CR and 3 for EQSI, an average of 8.8 persons in each group and an overall participation of 59% men and 41% women out of 158 total participants. For the mini-survey, a total of 115 respondents participated, with 58 for GP=SCPP, 38 for CR and 19 for EQSI for an overall participation of 71% men and 44% women.
In Phase 1 and 2, mini-surveys were given to farmers to self-administer following each FGD. In Phase 1 there were 148 mini survey responses out of 165 FGD participants, for an overall response rate of 89%. For CR, one FGD contained farmers who had not yet participated in training, so they were not administered the mini survey and this lowered the overall response rate.
In Phase 2 there were 115 mini survey responses out of 158 FGD participants for an overall response rate of 73%.
Survey instrument
The questionnaires for the project staff interviews were semi-structured guides that were strictly qualitative (open-ended), questionnaires also collected sex, title, and interview date and location. The questionnaire included perceptions of training approaches, government priorities, progression towards meeting indicator targets and sustainability.
Questionnaires were also delivered to staff at fermentation centers and cocoa buying stations to ask about post harvest handling and perceptions to cocoa quality during the course of the projects. The questionnaires were all developed and utilized in English and were translated to Bahasa where necessary (namely for government respondents of select KII).
A farmer survey focus group discussion guide was developed and verbally administered. Post-FGD farmers were given a paper, self-administered Mini-survey which was translated into Bahasa.
All questionnaires and tool are provided as external resources.
Data collection
| Start | End |
|---|---|
| 2017-09-18 | 2017-10-21 |
| 2019-10-07 | 2019-11-11 |
The Phase 1 ET members included a Team Leader (female), Cocoa Sector Specialist (male), Jr. Analyst (female) and Cocoa Research Coordinator (female). The Phase 2 ET members included a Team Leader (male), Cocoa Sector Specialist (male), Jr. Analyst (female) and Cocoa Research Coordinator (female). For both Phase 1 and 2, the team divided into two teams of two in the same selected district in order to minimize the risk of unprecedented issues being faced by a team member and to prevent bias. Each team had one Indonesian member to maximize local knowledge and language use and one team was accompanied by an interpreter. At each KII, FGD and survey administration, one team member led the interview/facilitated and the other took notes, accompanied by the interpreter. Recordings of direct observations were conducted by one Bahasa team member only. The two teams traveled by 4-wheel drive vehicle to all sites. The Team Leader (TL) was responsible for overall coordination of all evaluation activities including design of methodology and tools, keeping of confidential data, facilitating FGDs and KIIs, analysis and managing team member work assignments and presenting final results to MCC and MCA-I. The Cocoa Sector Specialist and Research Coordinator were responsible to assist the TL in design of evaluation, collect data and analysis, arrange logistics and inform report writing. The Junior Analyst was responsible to be the liasion between MCC/MCA-I and Social Impact, oversee quality assurance, collect and analyze data and report writing.
Phase 1- The ET conducted KIIs in English and Bahase and facilitated FGDs in Bahasa since most of the farmers are fluent and at least have completed primary school.
Phase 2- The ET conducted KIIs in English and Bahasa and facilitated FGDs in Bahasa since most of the farmers are fluent and at least have completed primary school.
For both evaluation Phase 1 and 2, the evalaution team collected data from Monday through Saturday. The ET took notes manually during data collection and typed them into a shared network drive daily. Additionally, all interviews were audio recorded with the permission of the interviewee, and the notes aided in transcription and analysis following each interview. Completed recordings were uploaded and saved securely on the Team Leader's external hard drive. For the mini survey, each team member distributed paper surveys to participants at the end of the discussion and read questions aloud to facilitate group participation. The ET member checked surveys for consistency. All data was entered into Microsoft Excel for analysis.
The instruments (KII guides, FGD guides, mini survey instrument and direct observation protocols) were translated into Bahasa and piloted in Jakarta and Makassar prior to field travel to check for comprehensibility for beneficiaries, logical flow and time required. Each data collection team consisted of one male and one female team member, and one team also included a male interpreter.
Data processing
Throughout site visits, the ET analyzed findings daily to determine emerging trends in order to aggregate findings around common themes. The ET used content and comparative analysis to identify response categories and patterns and identify emergent themes and contextual factors. After data collection, the ET aggregated data by coding interview responses and feedback obtained from the KIIs and FGDs around common and key themes related to the four EQs. For quantitative mini-survey data, the ET inputted data electronically into a working Excel document to conduct basic frequency tables for analysis and identify any emerging trends and compare subgroups via cross-tabulation. Data analysis tabulated responses and disaggregated data, where possible by project, private sector partner, region, and gender, to understand what changes occurred and how this might have varied among beneficiary groups. SI analyzed data obtained by FGDs by project and location to capture any differing perspectives of grant approaches and experiences among groups.
Access policy
Millennium Challenge Corporation
Millennium Challenge Corporation
https://data.mcc.gov/evaluations/index.php/catalog/206
Cost: None
Data Access
The dataset has been anonymized and is available as a Public Use Dataset. It is accessible to all for statistical and research purposes only, under the following terms and conditions:
- The data and other materials will not be redistributed or sold to other individuals, institutions, or organizations without the written agreement of MCC.
- The data will be used for statistical and scientific research purposes only. They will be used solely for reporting of aggregated information, and not for investigation of specific individuals or organizations.
- No attempt will be made to re-identify respondents, and no use will be made of the identity of any person or establishment discovered inadvertently. Any such discovery would immediately be reported to MCC .
- No attempt will be made to produce links among datasets provided by MCC, or among data from MCC and other datasets that could identify individuals or organizations.
- Any books, articles, conference papers, theses, dissertations, reports, or other publications that employ data obtained from MCC will cite the source of data in accordance with the Citation Requirement provided with each dataset.
- An electronic copy of all reports and publications based on the requested data will be sent to MCC.
The original collector of the data and MCC bear no responsibility for use of the data or for interpretations or inferences based upon such uses.
Social Impact. Indonesia Green Prosperity Sustainable Cocoa Partnership. Farmer mini-survey 2019. Millenium Challenge Coorporation 2020. https://data.mcc.gov/evaluations/index.php/catalog/206. Retrieved DATE.
Contacts
| Name | |
|---|---|
| Monitoring & Evaluation Division of the Millennium Challenge Corporation | mccopenmicrodata@mcc.gov |
Metadata production
DDI_IDN_2017-2019_MCC-GPCGP_v01_M
| Name | Role |
|---|---|
| Millennium Challenge Corporation | Review of Metadata |
2019-11-27
Metadata version
v02 (November 2019). This version is identical to v01 except that it has been updated to include edits in Phase 2 of the evaluation.
Version 3.0 (April 2020) - Including inputs from the Final Evaluation Report
Version 4 (July 2020). Edited version based on Version 3.0 (DDI-MCC-IDN-GP-COCOA-2020-v05) that was produced by the Millenium Challenge Corporation.
The GP Project identified and selected 13 provinces which were eligible for the GP Facility grants. Additionally, 24 districts within these provinces were identified by MCA-Indonesia as having favorable project development characteristics for the cocoa partnership grants. As the major cocoa growing region in Indonesia, the main geographic focus of the three cocoa grants is in Sulawesi. The SCPP grant conducts activities in four districts in East Nusa Tenggara, two districts in Southeast Sulawesi, two districts in South Sulawesi, two districts in West Sumatra and four districts in West Sulawesi. Cocoa Revolution conducts activities in one district in Southeast Sulawesi and one district in South Sulawesi, where it overlaps activity areas with SCPP. EQSI manages activities in three districts of Southeast Sulawesi, and does not overlap with either of the other grants. The ET sampled purposively at the province and district levels, targeting major cocoa producing regions with many participating farmers. The resulting sample includes most or all districts where CR and EQSI are implemented and four of ten where GP-SCPP is implemented, all either located in Sulawesi or Sumatra.
The study population includes farmers, buying station and fermentation center representatives, farmer cooperatives and associations, MCC/MCA-I project staff, grantees, field staff, goverment representatives, project management and private sector partners.