Procurement Modernization 2016, Independent Performance Evaluation
Indonesia, 2016
Get MicrodataIdentification
IDN_2016_MCC-PM_v01_M
Procurement Modernization 2016
Independent Performance Evaluation
| Name |
|---|
| Indonesia |
The evaluation seeks to establish evidence on the possible effects of an intervention designed to modernize public procurement: effects on cost savings, performance in the procurement process, and corruption, as well as economic growth. While previous MCC programs on reducing corruption in public procurement have been implemented in Paraguay, Uganda, and Kenya, generating interest in the potential of these programs to achieve results more broadly, these three prior programs have been threshold programs and relatively smaller in magnitude than the $50 million procurement modernization component of the Indonesia Compact. The Indonesia Procurement Modernization project is the first to try to achieve results broadly and at a national scale. The impacts of the Indonesia PM project evaluation may have implications for the design of future MCC programs, and with broader dissemination of its findings, may influence other donors as well. Although existing quantitative evidence on the introduction of e-procurement suggests that procedural and other reforms may improve procurement outcomes (Lewis-Faupel et al., forthcoming), there is little quantitative evidence on more- comprehensive reforms, and the MCC Procurement Modernization project evaluation seeks to fill this space with new and more compelling evidence. While we are not able to measure effects on economic growth as a result of the program directly, if the program is found to have strong effects on cost savings and procurement performance, a follow-on effect on economic growth may be plausible.
Sample survey data [ssd]
Administrative units (Procurement Service Units or PSUs)
Version
- v01: Edited, anonymous dataset for public distribution.
Scope
| Topic | Vocabulary |
|---|---|
| Anti-Corruption | World Bank |
| Administrative & Civil Service Reform | World Bank |
| Governance | World Bank |
| Financial Management | World Bank |
| Capacity Building and Institutional Strengthening | MCC Sector |
Coverage
The PM project is being implemented nationally, with the aim of establishing models of best practices across the country. While there was no specific effort to include every island, ensuring geographical diversity was a key objective PSU selection process.
The survey covered employees of PSUs and corresponding SKPDs (spending units) that are involved with the procurement process. Employees that have other roles at these institutions were not interviewed. Respondents are aged between 18-59 years and are 80% male.
Producers and sponsors
| Name |
|---|
| Abt Associates |
| Name |
|---|
| Millennium Challenge Corporation |
Sampling
26 PSUs were shortlisted to participate but 16 were selected to participate in the program. All Phase 2 treatment PSUs with the exception of the LKPP were surveyed for baseline. The comparison group for Phase 2 participants was created from those 10 ULPs that were shortlisted but not selected to participate in the project. As all these 10 PSUs had expressed an interest to participate in the project, the implicit selection bias is reduced by selecting them as our comparison group.
Within each PSU selected to participate in the quantitative survey (12 treatment PSUs and 10 comparison PSUs), we surveyed up to 20 PSU staff members along with up to 5 staff members at an associated spending unit. The selection criterion for the spending unit was based on the minimum cutoff number of contracts (atleast 5 contracts) it had with the PSU. For some PSU/spending unit pairsthere are more than five respondents from the spending unit because some respondents work at both the PSU and the spending unit. An effort was also made to have an even dispersion across the treatment and comparison ULPs of the type of spending unit interviewed, from the three most common types: public goods, education and health. The only Phase 2 PSUs that were not matched were those based in the ministries due to their unique nature. However, data was collected from the Ministry PSUs to allow for descriptive and pre-post analyses.
The original plan was to conduct 440 interviews, and 426 were successfully completed for a 97% response rate. Of these 426, 295 work exclusively in the PSU, 75 work exclusively in the spending unit and 56 work in both the ULP and the spending unit.
Three spending units that had originally been selected to be interviewed were changed due to security reasons, and communication issues/unresponsiveness from PSUs. The replacement spending units were selected from a list proivded by Abt of preapproved alternatives should any issues arise during the data collection.
The original plan was to conduct 440 interviews, and 426 were successfully completed for a 97% response rate. Of these 426, 295 work exclusively in the ULP, 75 work exclusively in the spending unit and 56 work in both the PSU and the spending unit.
Observations in this data set use a finite population correction for analysis: the response rate of the observation's respective place of employment. Because different combinations of respondents (ie. PSU staff vs. spending unit staff vs. both) responded to different sections of the survey, the fpc varies depending on which questions are being analyzed.
The variables used as the fpc are:
- For questions asked of everyone: samp_rate_both
- For questions asked of PSU employees only: samp_rate_psu
- For questions asked of spending unit employees only: samp_rate_sp
Survey instrument
Abt Associates has designed an original baseline survey instrument to measure outcomes of interest to the PM evaluation. This survey instrument contains 10 modules and covers a broad range of topics, including administrative structure/staff professionalization, procurement practices, desirability of procurement career paths, involvement along the procurement continuum, procurement timeliness and efficiency, framework contracting/e-catalog/PMIS/PPP, use of performance monitoring data, perceived levels of bias and collusion, and procurement knowledge.
The questions were developed in English and were translated into Bahasa Indonesian. After an initial review the questionnaire was translated back into English by an independent translator with no prior knowledge of the survey.
Data collection
| Start | End |
|---|---|
| 2016-09-14 | 2016-10-07 |
| Name |
|---|
| SurveyMETER |
For the survey, there were a total of 11 field teams, each consisting of one supervisor and three enumerators. Thus the survey required a total field staff of 44 people: 11 supervisors and 33 enumerators. There was an assistant team leader (ATL) who is responsible for assisting the team leader in the day to day management of the survey and recruitment, training and coordination of the field teams. The ATL was in regular contact with the field teams to clarify confusions over concepts or definitons and to provide technical assitance.The supervisors were responsible for checking the completed questionnaires for plausibility and consistency, verifying the results of interviews, and managing the planning and logistical aspects of the survey teams.
Training lasted for 6 days, the first three of which were dedicated to understanding the design and content of the questionnaire as well as how to use the CAPI program.
For the survey, there were a total of 11 field teams spread over the different islands, each consisting of one supervisor and three enumerators. Thus the survey required a total field staff of 44 people: 11 supervisors and 33 enumerators.
Data collection took around 2 weeks in September of 2017, during which time the enumerators conducted interviews every day when possible. The interviews were conducted in Indonesian.
Data processing
The Abt team developed a Stata do file to quality check the data every day when it was received from the survey firm. The do file included the following checks:
- checked for missing identifiers
- checked for any unrealistic quantities in numeric variables
- checked to ensure skip patterns built into the electronic survey instrument were working correctly
- checked for patterns by surveyor (different responses for the same question, number of "don't knows" or "refused", quiz scores within the survey as compared to the average, consistency in responses)
The file that produces this analysis data set appropriately codes missing variables, including coding "dont knows", "refused" and "not applicable" as ".d", ".r", and ".n" respectively. In addition, it creates a few new variables including the quiz grades from the quiz section and the variables used for the finite population correction.
Access policy
Millennium Challenge Corporation
https://data.mcc.gov/evaluations/index.php/catalog/188
Cost: None
Data Access
Use of the dataset must be acknowledged using a citation which would include:
- the Identification of the Primary Investigator
- the title of the survey (including country, acronym and year of implementation)
- the survey reference number
- the source and date of download
Contacts
| Name | |
|---|---|
| Monitoring & Evaluation Division of the Millennium Challenge Corporation | impact-eval@mcc.gov |
Metadata production
DDI_IDN_2016_MCC-PM_v01_M
| Name | Role |
|---|---|
| Millennium Challenge Corporation | Review of Metadata |
2020-06-17
Metadata version
Version 02 (June 2020). Edited version based on version 1 (DDI-MCC-IDN-PM-2016-v1) that was done by the Millennium Challenge Corporation.
The survey covered employees of PSUs and corresponding SKPDs (spending units) that are involved with the procurement process. Employees that have other roles at these institutions were not interviewed. Respondents are aged between 18-59 years and are 80% male.