Vocational Education Project 2018, Independent Performance Evaluation

1.1.1 Contribution vs Attribution
The evaluation design focuses on ex-post identification of areas of change and exploring mechanisms through which changes occurred through largely qualitative data, with substantiation through quantitative data when available. Accordingly, the absence of a valid counterfactual against which to compare intervention effects limits the ET's ability to determine attribution of observed effects to VEP inputs alone. Additionally, when a new administration was elected near the end of the Compact, significant changes were made within the TVET system, including the dissolution of the TVET advisory council that included private sector participation. These changes present confounding factors that further complicate the ET's ability to disentangle the independent effects of VEP. The PE will, however, provide an opportunity to assess the contribution of the VEP to important changes and identify and assess potential mechanisms of change, which could inform future projects and evaluations in the TVET sector.
1.1.1 Accessing Contact Information
This PE relies on the availability of contact information for beneficiaries and VEP participants and a variety of secondary data. First, regarding beneficiary and stakeholder contact information - Uncertainty surrounding the ET's ability to access key individuals due to the five-year gap between Compact close and this PE poses a challenge to the proposed evaluation design. The ET will mitigate this risk by first working closely with MCC to collect and/or identify contact information. The ET will also use local ET members to establish strong support in-country to mediate these discussions with local respondents and support fieldwork scheduling. As noted above, the ET also plans to utilize snowball sampling during fieldwork to help fill in any gaps left in the sampling frame after coordination with the EMC. Second, regarding secondary data, the ET will request secondary datasets through MCC. Any direct communication with the GoM will be coordinated through appropriate MCC channels, if available. For this task, strong support in-country from a local ET member will also be critical.
1.1.1 Recall and Response Biases
The ET acknowledges two inherent biases associated with the proposed qualitative data collection. One limitation is the possibility of recall bias among key informants. The ET will take steps to reduce recall bias in the data collection tool design phase, including framing questions to aid accurate recall. Where possible, the ET will corroborate interview findings with additional data sources, such as GoM records. Potentially relevant records include enrollment and graduation statistics for the country as a whole plus records on PPPs and other types of private sector engagement. The ET also acknowledges the potential for bias due to respondent subjectivity and the possibility of collecting only socially desirable responses from interviewees. To address this potential bias, the ET will purposively recruit a diverse sample of informants and triangulate responses with other data sources, all while developing data collection tools based on best practices that minimize response bias. Since the ET will not be able to avoid all bias in the data, persistent biases will be noted, where applicable, in the discussion of results of the final report.