DDI_TLS_2014_MCC-T_v01_M Millennium Challenge Corporation 2015-01-05 NADA Version 1.0 (January 2015) Version 2.0 (May 2015). Edited version based on Version 01 (DDI-MCC-TLS-SI-THRESHOLD-2014-v1) that was done by Millennium Challenge Corporation. Threshold 2014 MCC-T 2014 TLS_2014_MCC-T_v01_M Social Impact NADA Millennium Challenge Corporation Monitoring & Evaluation Division Independent Performance Evaluation Not applicable to this evaluation; no quantitative data to be shared Transparency Reform Corruption Capacity Building and Institutional Development This final performance evaluation is intended to provide an objective, rigorous, and evidence-based analysis of the Fostering Transparency Initiative's (FOTI) overall performance. Specifically, the evaluation will look to identify, review, and assess four key aspects by utilizing the following evaluation questions: 1. Program Logic a. Were the original six program components relevant to the Timorese context? (e.g. were they based on initial analysis of assumptions, risks, and external factors?) b. Were any changes made to the program logic during the implementation? If so, were they appropriate? 2. Activities Designed and Implemented a. Were planned activities actually implemented? Why or why not? b. Were any new activities designed and implemented? If so, why? c. What was the effectiveness of activities? 3. Influence on Changes in Knowledge, Attitudes, and Behaviors a. Did implemented activities lead to measurable outcomes, such as changes in knowledge, attitudes, or behavior? 4. Sustainability & Lessons Learned a. Have key implemented activities been sustained? b. What is the likelihood for future sustainability? c. What are the key aspects of FOTI that should be continued in the near term? d. Which aspects of FOTI would be appropriate to replicate in other contexts? Timor-Leste National coverage Individuals Other Casals & Associates Social Impact, Inc. A random sample of monitoring data from 17 trainings was analyzed to assess both the strength of the technical questions in the tests and also the level of trainee learning as captured by the post-tests. The Team Leader interviewed also 14 key informants interviewed during the preliminary fieldtrip in February 2014 using convenience sampling. A total of 44 informants were interviewed by the full field team during the field visit in July 2014. Four additional stakeholders were interviewed via phone after the fieldwork was completed. The mixed methods evaluation was carried out by a four-member team from Social Impact, Inc. during field visits in early 2014. In February 2014, while the FOTI project was still active, the Team Leader completed preliminary data collection during an evaluation design trip. In July 2014, the other three team members completed data collection during a two week field visit to Timor-Leste. The evaluation utilized ongoing data analysis, an iterative methodology, and careful triangulation of data sources throughout the research process. The evaluation team utilized the following methodologies: document review; technical evaluation of "deliverables"; pre- and post-test data; key informant interviews; and a field visit to Baucau. Although the evaluation team attempted to complete a rigorous examination of the approved research questions, it proved impossible to utilize all of the proposed research methodologies due to logistical and bureaucratic realities in Timor-Leste. For example, although the team initially planned to conduct mini-surveys and focus group discussions, the team did not gain access to enough program beneficiaries to make data obtained from such informants meaningful. Other limitations include: challenge in measuring qualitative outcomes; logistical and timing challenges; recall and attribution bias; selection bias; and use of interpreter. For further details, please see Annex II. While these factors limit the quantity and quality of the data collected, the team made every effort to fill gaps in the data, particularly by examining all available program documentation. Through this triangulation of sources, the team feels that the information presented in this report is an accurate depiction of the FOTI program. Millennium Challenge Corporation Millennium Challenge Corporation http://data.mcc.gov/evaluations/index.php/catalog/136 Cost: None U.S. Agency for International Development. Final Performance Evaluation of the Fostering Transparency Initiative Program in Timor-Leste. September 2014. The report is available for USAID/MCC internal review and is not open to the public.