There is a lack of evidence on both the development impacts of rural transport
improvements and their benefits to the poor. This knowledge gap stems from the
methodological weaknesses of existing SSA impact studies and the failure to undertake
robust baseline data collection before the launching of a rural transport
project. This failure is compounded by poor sample design and analysis of collected
data. Impact studies tend also to be traffic focused and assume that resultant
transport services are affordable and appropriate to the poor.
In project terms, impact evaluations are the final stage of the M&E process, preceded
by and to some extent drawing on the results monitoring of the rural
transport project outcomes. Impact evaluation is therefore a post project activity
used to assess whether the investment has achieved its development goal.
Most rural transport projects use quantitative techniques to assess impacts, which
belong to five main types. The first two are macro and sector studies, using secondary
data to test existing theories and hypotheses on the relationship between
rural transport and the development process, and predict the likely poverty reducing
impact of a rural transport investment policy.
The next three are used to evaluate specific projects. They range from cross sectional
studies to panel surveys with the latter emerging as the more robust and
methodologically sound approach. This is particularly the case if a Randomized
Control Trial (RCT) sampling of household respondents is adopted and propensity
score matching is applied to identify comparable treatment and non-treatment
groups as the basis for the counterfactual—what would have happened without
the intervention. These have been called the “gold standard” impact methodologies
but rural transport projects may not be large enough to warrant the technical
and financial resources needed to collect and analyze the large quantities of impact
data required by such an evaluation. This emphasis on increasingly sophisticated
quantitative techniques has meant that qualitative techniques are usually used to triangulate or crosscheck the econometric findings. Rarely are they used as a
stand-alone evaluation.
All of these techniques use a range of indicators to measure direct and indirect
effects and impacts. One indicator though missing from this list is the Rural Access
Indicator (RAI), the defined percentage of population living within 2 kilometers
(20-25 minutes’ walk) of an all-weather road. This type of data is usually collected
and developed at the macro level but there are notable inconsistencies in the way it
is measured between different SSATP member countries. These inconsistencies
need to be addressed if the RAI is to remain as a high-level access indicator capable
of generalization within and between countries.
The experiences of impact studies carried out by two SSATP country members
have been reviewed and this has identified a number of key principles that should
be followed before a RT project commits to undertake an impact study:
1. There has to be a strong government and sector interest and commitment to
undertake a robust impact evaluation. The financial and capacity needs of this
commitment lie beyond the scope of most SSA countries and it is expected
that development partners’ support will be needed to address this shortfall.
2. Ideally, the impact study should be also aligned with government systems and
be part of a capacity building exercise to carry out evaluations as part of the
normal administrative, sector and governance functions that asks “what
works, and for whom” This widens to impact evaluation to include country
statistics offices (CSOs) and where appropriate university staff/research institutions.
It should be possible to draw expertise from these organizations and
put together a team of national researchers, supported by one or more international
experts who have experience of conducting robust road impact evaluations
outside SSA.
3. The scope of the evaluation has to be clearly defined by the client (public authorities).
It should answer one or more of the following questions:
- Is the intervention making a difference – What has happened because
of the intervention?
- What are the results on the ground – Has the project delivered its expected
benefits?
- These in turn can be addressed by a number of indicators or variables,
which can be collected by both quantitative and qualitative techniques,as a baseline before the project has initiated any works and at various
stages after the works have been completed.
- The cost of impact studies reflects the scope defined by the client. At its
simplest, an evaluation can undertake qualitative user-focused access
surveys costing $100/200,000. At the other extreme, “gold standard”
randomized control trials costs can exceed US$1 million in today’s
terms. The costs of the latter, and the need for international expertise,
make them suitable only where there is significant government and DP
commitment to the rural transport sector.
The adopted methodology for an impact evaluation is usually a compromise between
the information needs of the funding agency/counterpart line ministry and
the project provision for M&E. Budget constraints usually mean that project management
focuses on methodologies that quantify project results or outcomes notably
transport cost savings enjoyed by road users. Thus, many rural transport project
logical frameworks specify traffic or access changes as objectively verified indicators
with a supplementary expectation that there will be a similar reduction in
transport charges. Access indicators associated with attendance and use of markets,
health centers, schools, etc. might also be included as objectively verified indicators.
These performance or outcome indicators are easy to collect and analyze
while project management and the subsector ministry alike will readily understand
the findings and justify the investment in cost benefit terms without any need to
understand the long-term impact.
This approach has worked where the road network is well trafficked, which is not
the case for a number of rural transport projects. In this situation, the proponents
of quantitative techniques have argued that traffic-based evaluations need widening
to include household and community surveys, which capture the full development
impact of RT improvements.
The high cost and resource demands of this more rigorous and defensible impact
study mean that it is only occasionally used, usually in situations where development
partners and clients have a need for impact data to inform their policy and
program commitments. Most rural transport projects usually adopt a results or
performance approach to project monitoring. This type of impact evaluation is
methodologically sound if it stays focused on the direct traffic and transport benefits
of a project. In this way, it provides subsector feedback on the success of its
planning and appraisal procedures and meets the accountability needs of financing
agencies and development partners. However, it has a number of weaknesses the
Rural Transport in Africa – Monitoring & Evaluation
xvi
most important of which is its inability to assess the distribution of benefits in
poverty terms. Here it is recommended that qualitative PRA techniques in social
mapping/modeling and wealth/well-being ranking are explored as a means of answering
the question of who benefits from rural transport interventions.