The key research questions guiding our design of the RRRP evaluation are:
• Did rehabilitating roads affect the quality of roads?
• Did rehabilitating roads improve access to markets and social services?
• Did rehabilitating roads affect agricultural productivity and profits, and if so, by how much?
• Did rehabilitating roads improve household well-being for communities served by these roads, especially income and poverty?
As is frequently the case for large scale infrastructure projects, it was not feasible to randomly select the roads that would be rehabilitated in the MCA RRRP from amongst the eligible. Instead, MCA planned to fund projects based on estimated economic rates of return (ERRs). The ERR is calculated from several inputs, including the vehicular traffic, vehicle operating costs, and the cost of the project, among others. An ERR was estimated for each of the road links under consideration for rehabilitation, and in order to be funded, a road’s ERR had to meet or exceed 12.5 percent. However, the MCC Board subsequently made the decision to discontinue funding any further road construction and rehabilitation under the Compact due to concerns about Armenia's democratic governance. Starting in 2009, the Armenian government instead accessed loans from the World Bank to rehabilitate many road sections that were included in the RRRP plans before MCC discontinued funding, using and updating the road project designs developed by MCA-Armenia. These roads were selected based on estimated ERRs as well as other factors such as the size of the population who would benefit.
The evaluation of rural road rehabilitation uses a comparison group design. The evaluation design identifies the counterfactual by defining a comparison group of roads selected from those initially proposed by MCA that share similar characteristics as those selected by the World Bank, but that were not rehabilitated. The evaluators adopted a hybrid approach that matches road links based on a limited number of key observable characteristics and then uses regression modeling to control for other characteristics that may impact outcomes. The most fundamental criterion used in identifying potential comparison roads is that it is among the roads that were originally considered for rehabilitation by MCA, indicating that the road conditions are poor enough to consider rehabilitation.
The data for this evaluation come from Armenia’s Integrated Living Conditions Survey (ILCS), an annual household survey fielded by Armenia’s National Statistical Service (NSS). MCA funded a considerable increase in the rural sample, which was in part aimed at facilitating the RRRP evaluation. The sampling plans were developed and implemented based on the evaluation design described above. The evaluation includes households served by 27 roads that were rehabilitated with World Bank financing based on designs funded by MCA and 28 comparison roads.
Kind of Data
Sample survey data [ssd]
Unit of Analysis
Outcomes are measured at the household level.
The analysis covers households in 107 communities served by 55 road links spread throughout the 10 regions of marzes (excluding the capital city Yerevan).
Unit of Analysis
Outcomes are measured at the household level.
The analysis covers households in 107 communities served by 55 road links spread throughout the 10 regions of marzes (excluding the capital city Yerevan), and the sample included in the analysis does not, in a statistical sense, generalize to a larger set of a communities.
The ILCS is a nationally-representative survey of Armenian households. Additionally, to accommodate the road rehabilitation and associated evaluation plans, a special sample covering 82 planned road links was drawn. However, these were ultimately not able to be used in the evaluation and will not be covered in the public use file.
Producers and sponsors
Authoring entity/Primary investigators
Mathematica Policy Research
National Statistical Service of Armenia
Goverment of the Republic of Armenia
Fielded and cleaned the Integrated Living Conditions Survey
Millennium Challenge Corporation
The core sample of the ILCS includes 768 enumeration areas, each containing 8 households for a total sample of approximately 6,100 households. Additionally, MCA funded an oversample of 216 enumeration areas in rural communities, an increase of approximately 1,700 households. The oversample is dedicated exclusively to communities served by roads in MCA's initial set of rehabilitation-eligible roads. Additional communities served by rehabilitation-eligible roads were selected into the core sample by chance, and a total of approximately 2,200 households served by the original, eligible project roads are in the sample each year. Because the ILCS does not use the same sample of enumeration areas in each year of the survey, different rounds of the survey may include different communities served by these roads.
The treatment group in our analysis comprises communities served by the 27 road links that were in MCA-Armenia's original RRRP plans, were ultimately rehabilitated in 2009 and 2010, and were covered by the ILCS from 2007/2008 to 2011. Unfortunately, ILCS data were unavailable for the relatively small number of communities affected by MCA-funded rehabilitation of the pilot roads. As a result, all 27 treatment links in the evaluation sample were funded with World Bank loans to Armenia. The comparison group in our analysis comprises communities served by the 28 road links that were in MCA-Armenia's original RRRP plans, were not ultimately rehabilitated with financing from the World Bank as of 2011, and were covered by the ILCS from 2007/2008 to 2011. Our data include outcomes measured before any roads were rehabilitated and after they were completed for both the treatment and the comparison group road links. All told, our analysis sample includes over 50 communities and 2,300 households in each of the treatment and comparison groups, for a grand total of 107 communities (55 in treatment and 52 in comparison) and 4,848 households (2,560 in treatment and 2,288 in comparison).
We use sampling weights constructed by the National Statistical Service that accounted for the number of households sampled and the population of communities surveyed in each road link. The total weight across households for each road link is the same for all links. Household weights were rescaled to give each road link equal weight. The exceptions are when two links are actually connected and serve the same set of communities, in which case those links are combined into road link pairs. In such cases the road link pair has double the weight of individual road links.