Conservancy Support and Indigenous Natural Products 2011-2014, Independent Performance Evaluation
Namibia, 2011 - 2014
Get MicrodataIdentification
NAM_2011-2014_MCC-CSINP_v01_M
Conservancy Support and Indigenous Natural Products 2011-2014
Independent Performance Evaluation
| Name | Country code |
|---|---|
| Namibia | NAM |
The evaluation employs a mixed-methods approach in which qualitative techniques and quantitative analysis support each other, recognizing that the techniques used will depend on the evaluation question to be addressed.
The source of information for the qualitative analysis is through Key Informant Interviews (KIIs) and Focus Group Discussions (FGDs) with the household and conservancy or PPO member-households and management, as well as with stakeholders in the tourism sector from the private-sector and associated regulatory bodies.
In the case of the quantitative analysis, control groups are not available for the evaluation of either the CS activity or INP sub-activities. The CS activity is taking place in most of the conservancies of the Northern Communal Areas (NCAs), which were selected for their tourism potential. Conservancies outside of this activity are generally in areas with differing natural endowments and market access and, as such, cannot serve as a comparable set of non-intervention conservancies. In the case of the INP sub-activities, it is not feasible to establish a valid comparison group because the intervention covers nearly the entire INP producer population. Instead, a type of a reflexive (before-and-after) design called a dose-response model is employed whereby each conservancy or PPO at baseline contributes to our understanding of the counterfactual by allowing us to infer whether differences in the amount of Compact assistance (the “dosage”) influence and, therefore, impact on CS or INP performance.
The model identifies likely program impacts by estimating the marginal effects of different intervention levels (e.g., intensity of training or number and type of grants) on outputs and outcomes of interest at critical points along the causal chain from the short to medium run. Originally, program impact on household income, the ultimate expected result by MCA-N, was to be a focus of examination, but it is now accepted that such changes would not likely be large enough to be detected over the relatively short evaluation period.
Sample survey data [ssd]
Depending on the research question: conservancy, PPO, household, individual
Household survey: The primary unit of analysis is the household. The definition of a household for the purposes of this survey is a group of people that live in the same compound and take meals together at least 4 days a week, as well as young children living elsewhere that are answerable to the head of the household. Several questions in the questionnaire apply to each individual family member.
Version
Anonymized dataset for public distribution
Scope
| Topic | Vocabulary |
|---|---|
| Other | MCC Sector |
| Namibia |
Coverage
Evaluation: Northern Communal Areas of Namibia
Household Survey: The survey covers twenty-three INP producer organizations and twenty-eight conservancies in nine regions of Namibia. It covers the conservancies and INP producer organizations of interest and is not meant to be nationally representative.
For Conservancies, there are a total of 76 conservancies in Namibia, 28 of which are included in the evaluation.
For INP, the sample frame only included membership from 28 PPOs (out of 63 targeted PPOs) since only the most organized PPOs could provide membership lists.
Producers and sponsors
| Name |
|---|
| NORC at the University of Chicago |
| Name | Affiliation | Role |
|---|---|---|
| NORC at the University of Chicago | External evaluator | |
| Survey Warehouse | Subcontractor to NORC | Local Data Collection Firm |
| Name |
|---|
| Millennium Challenge Corporation |
Sampling
CS/INP Survey
For the CS Activity, the baseline sample design was a two-stage sample design in which the first-stage primary sample units (PSUs) were Enumeration Areas (EAs) that overlapped with conservancies and the second-stage sample units were households. A number of variables were known (from Census and GIS sources) for each EA (or the constituency in which an EA was located) that could be used to assist sample design. To select households within each PSU, two random starting points were selected, and six households were selected for each random starting point (from each starting point, every fifth household was selected). This methodology was carried out because NORC did not have lists of households for each conservancy. NORC therefore did not have a sample of households before starting fieldwork; instead Field Interviewers (FIs) selected households based on a step-wise method using a systematic walk pattern from the random starting point. When a household was absent, interviewers went to the next available household.
The endline sample for the CS Activity consisted of a 100% panel sample with replacement. In other words, FIs re-contacted all baseline respondents. If a baseline respondent could not be interviewed (refusal or unlocatable), FIs replaced the respondent with the next available household to the right of the respondent's home.
For the INP activity, the sample design was comprised of households selected from lists of PPO producers (which includes households both inside and outside conservancies). The sample design was originally going to be a two-stage design in which the first-stage sample units were communities on the PPO list (e.g. villages) and the second-stage sample units were households within the selected communities. Unfortunately, it was not possible to obtain community locations for many of the producers in the INP sample frame, so it was not practical to implement the original concept of selecting a two-stage sample for the INP survey. In fact, few variables were available in the PPO frame that could be used to construct an analytical survey design. Apart from PPO, the only variable useful for constructing an analytical design was INP species. Therefore, it was decided to select a stratified single-stage sample from the frame, where stratification would be by INP species.
While the target sample size for the INP household survey was 500 households, the final baseline dataset only includes 296 household interviews. As a result, in addition to the 296 households from the baseline survey, the endline INP household survey also includes an additional 204 INP households in order to reach the original target of 500 households and to provide additional data points for the endline analysis. These additional households were sampled from the original baseline sample frame in order to maximize comparability. It is important to note the following points concerning this additional sample:
(1) Of the 28 PPOs represented in the baseline sampling frame, 18 PPOs are represented in the final baseline dataset. Because NORC does not have baseline information for the PPOs which were not surveyed at baseline, including them at endline would not be useful for the CS/INP evaluation given that a pre-post analysis would not be possible. Therefore, the additional 204 endline households were drawn from the 18 PPOs represented at baseline only.
(2) However, the 18 PPOs represented in the baseline dataset do not cover Commiphora. Given that Commiphora is one of the main INPs targeted by the intervention, the final endline sample also included Commiphora PPOs (in addition to the 18 PPOs included at baseline). While a pre-post analysis would not be possible for these harvesters, the data from these harvesters can be used to generate descriptive statistics about the endline period.
CS/INP Focus Group Discussions
For the CS Activity, a total of 40 FGDs were conducted with conservancies. For the midline data collection, 12 conservancies were selected to cover a wide range of conservancy characteristics such as geographic location, size, population and institutional level. For each conservancy, two focus groups were conducted: one with the members of the management staff and one with members who did not hold a management position with the conservancy, bringing the total of midline CS FGDs to 24. For the endline data collection, 8 of the original 12 conservancies were re-selected and similar to midline, FGDs were conducted with management and non-management members, bringing the total of endline CS FGDs to 16.
For the INP Activity, a total of 40 FGDs were conducted with members of PPOs. For the midline data collection, 12 PPOs were selected to cover a wide range of PPO characteristics such as geographic location, implementer, and institution type. For each PPO, two focus groups were conducted: one with the members of the management staff and one with members who did not hold a management position with the PPO, bringing the total of midline INP FGDs to 24. For the endline data collection, 8 of the original 12 PPOs were re-selected and similar to midline, FGDs were conducted with management and non-management members, bringing the total of endline INP FGDs to 16.
Recruiting Focus Group Discussion participants was mostly done with the help of the senior staff of the conservancies and the PPOs, typically either chairpersons or coordinators whose contact information was received from the relevant implementer. The teams made phone contact prior to arriving at conservancies or PPO areas and made arrangements to meet upon arrival. All recruitment criteria and procedures were discussed in these first meetings. Conservancy/PPO representatives then listed the names and locations of the appropriate respondents. Where possible, with the help of the chairpersons or coordinators, the field team made phone calls to the identified respondents to make arrangements to meet and discuss the study and invite them to participate. In other cases, where time allowed, the chairperson or coordinator organised a meeting for the team to meet with and conduct all necessary arrangements with a pool of potential respondents at a central location. From this pool, respondents were then selected as per the recruitment protocols.
CS/INP Key Informant Interviews
Sample development for the key informant interviews (KIIs) was a joint effort between the NORC, MCA-N, and Survey Warehouse teams. NORC produced a list of different types of respondents to potentially pursue, and the conservancy and INP experts on the evaluation team refined the list and suggested names of individuals in some categories. After this list was shared with MCA-N, NORC met with MCA-N staff to discuss the list, further refine it, and obtain contact details for potential respondents. The list of potential KII respondents continued to evolve throughout the recruitment process.
For the CS Activity, a total of 20 CS KIIs were conducted, 8 during the midline round and 12 during the endline round.
For the INP Activity, a total of 19 INP KIIs were conducted, 12 during the midline round and 7 during the endline round.
CS Survey. At baseline, 1032 interviews were completed from an initial target of 1188 interviews (although NORC was contractually required to complete 1000 interviews). When a household was absent, interviewers went to the next available household but did not document the households that they attempted to interview and which were unsuccessful. This oversight made it difficult to calculate an accurate response rate based on the total number of attempted interviews. However, there were 99 EAs in the sample and a target number of 12 interviews in each EA. The total target sample size was thus 1188 (12 x 99). We can use this target sample as a proxy for calculating the response rate. With these assumptions, the response rate would be: 1032/1188 = 86.9%.
For the endline CS survey, 1024 interviews (out of a target of 1032) were completed. Of these, 241 are replacement households; therefore, field teams managed to re-interview 76.5% of respondents.
INP Survey. At baseline, 298 interviews were completed out of 631 attempted interviews. NORC was asked to complete 500 total INP surveys. However, given the problems with the initial frame, it was impossible to reach this target. The overall response rate based on the households within scope is: 298/502 = 59.4%.
For the endline survey sample there were two different “pools” of people: a pool of 296 respondents which were interviewed at baseline, and a pool of 204 "new households" interviewed at endline, for a total target of 500 respondents. Where INP households could not be located, supervisors used the sample replacement list provided by NORC to locate the next replacement. Overall, 496 interviews were completed against a target of 500. A total of 143 replacements were made.
Harvester Sampling Weights for Baseline and Endline by INP:
The baseline sample is constructed from 18 PPOs out of approximately 63 that were operational and had member lists in 2009. The number of households selected was established by MCA prior to NORC's participation. Harvesters were randomly selected from a sampling frame comprising of the PPO member lists of the subset of PPOs specializing in the respective INP. With this in mind each weight is calculated as the reciprocal of the probability of selection. This requires dividing the population size by the actual baseline or endline sample size. The use of these weights depends on whether one is conducting panel analysis or cross-sectional comparisons. For panel analysis the baseline weights is applied to both rounds. For comparison of discrete cross-sections, the baseline and endline weights in the table is applied accordingly. The reason for this difference is that the latter includes additional observations randomly sampled from the same population so the larger sample implies that each observation.
Baseline weights by INP:
- Devil's Claw: 14.226
- Marula: 10.621
- Ximenia: 5.814
Endline weights by INP:
- Commiphora: 5.825
- Devil's Claw: 5.765
- Marula: 10.191
- Mopane: 5.333
- Ximenia: 5.732
Conservancy Household Sampling Weights for Baseline and Endline:
For the sampling plan, NORC generated 90 site-specific sampling weights for about 1,000 households - that is, on average about twelve households receive the same weight. A complex process was used and involved a technique called variable stratification. The CS survey was a two-stage sample in which the first-stage sample units (primary sample units, or PSUs) were portions of Census Enumeration Areas (EAs) that overlap with conservancies, and the second-stage sample units (elements, ultimate sample units) were households. The first-stage sample units (EA portions) were selected with variable probabilities, to assure adequate variation in explanatory variables related to outcomes of interest. The second-stage sample was a random sample of households selected from each first-stage sample unit. The intended household sample size within each PSU was 12, but this varied a little, because of inability to obtain this number of sample households in a PSU, or to obtain replacement households for PSUs falling short of this target number.
An “unnormalized” sample weight is obtained for each PSU by dividing the total number of households in each sample PSU by the product of the PSU selection probability times the number of sample households in the PSU. The term “unnormalized” is used because of sampling variability. A “normalized” per-household weight is obtained by normalizing the resulting values such that the total weight for all sample households equals the total number of households in the population.
Since the endline replacement households were “near” to the original household, the sampling weight from any lost household was also applied to the household that replaced it.
Survey instrument
2013-2014 Qualitative Data Collection:
The conservancy guide covers the following topics, as relevant:
- Business partnerships and conservancy revenue
- Conservancy governance
- Effect of game acquisitions
- Household well-being
- Gender dimensions of access and benefits
- Sustainability
The PPO guide covers the following topics:
- PPO organisational capacity
- Harvest, sales and income
- Household wellbeing
- Intra-household gender considerations
- Sustainability
2011-2014 Quantitative Data Collection:
-
Household surveys. Two rounds of the CS survey and the INP survey, which are explicitly designed for the evaluation, and will track the same group of 300 INP harvester-households and 1,000 CS members in 2011 (baseline), and 2014 (endline). To compensate for INP harvesters not accessible during baseline an additional 200 will be interviewed at endline to bring the endline total to 500. These data will provide information on important measures of impact, as well as on household characteristics and demographics.
-
Organisational surveys. Both to track governance and management improvements at the level of the conservancy and PPO, as well as to collect fixed-effect covariates for household-level multivariate analysis, “factsheets” will be completed for each conservancy and PPO. In the case of the former, NORC will draw on the implementer databases (see next bullet) as well as the 2009 ARD baseline conservancy needs assessment (CNA) of conservancy institutional capacity. In the case of the PPOs, the expectation is that the factsheets could be completed using NRI's database of monitoring data; any remaining unanswered questions would be answered during the fielding of the household survey.
-
Implementer databases. For the CS activity these would include the NACSO Community-Based Natural Resource Management (CBNRM) database, which has annual information on key economic indicators of interest such as revenues at conservancy level and share of conservancy revenue paid out in dividends, as well as conservancy-level GIS data and game counts available through internal databases for the Conservancy Development Support Services (CDSS), and the Conservancy Development Grants Fund (CDSGF). Separately, there are data on the size of grants and the geographic distribution of services and grants. For the INP activity these would include Natural Resources Institute (NRI), University of Greenwich's program monitoring outputs.
Data collection
| Start | End | Cycle |
|---|---|---|
| 2011-07 | 2011-07 | Baseline |
| 2013-07 | 2013-08 | Midline |
| 2014-04 | 2014-05 | Endline |
Baseline (household survey): July 2011 Midline (focus groups and key informant interviews): July-August 2013 Endline (household survey, focus groups and key informant interviews): April-May 2014
| Name |
|---|
| Survey Warehouse |
Data for the household survey was collected via face-to-face paper-and-pencil interviews (PAPI) in respondents' homes. Separate versions of the questionnaire were used for the conservancy members and harvesters of indigenous natural products (the two questionnaires differed only in a handful of questions).
Data processing
NORC performed a number of logic and consistency checks on the data.
Access policy
Millennium Challenge Corporation
Millennium Challenge Corporation
http://data.mcc.gov/evaluations/index.php/catalog/152
Cost: None
Data Access
Zinnes, Clifford, Kareem Kysia, Alejandro Ome et al. 2014. Evaluation of MCA Namibia's Conservancy Support and Indigenous Natural Products Activities. Chicago, IL: NORC at the University of Chicago.
Contacts
| Name | Affiliation | |
|---|---|---|
| Monitoring & Evaluation Division | Millennium Challenge Corporation | impact-eval@mcc.gov |
Metadata production
DDI_NAM_2011-2014_MCC-CSINP_v01_M
| Name | Role |
|---|---|
| Millennium Challenge Corporation | Review of Metadata |
| NORC at the University of Chicago | Metadata Producer |
2015-02-26
Metadata version
Version 1.0 (February 2015)
Version 2.0 (April 2015). Edited version based on Version 01 (DDI-MCC-NAM-IE-AG2-2014-v01) that was done by Millennium Challenge Corporation.
Version 3.0 (July 2020). This version adds a description of the collected data.
CS Activity. The CS Activity focused building management capacity at 31 conservancies.
INP Activity. Approximately 7,000 primary producers, and management from 60 PPOs. Other organizations were also able to compete for funding through the INP Innovation Fund.
Household survey: Members of the target conservancies and harvesters belonging to producer organizations focused on the indigenous natural products of interest.