External Evaluation of BOTA Programmes: The Impact of BOTA's Conditional Cash Transfer (CCT) Programme 2011-2012
Kazakhstan, 2011 - 2012
Get MicrodataIdentification
KAZ_2011_BOTACCT_v01_M
External Evaluation of BOTA Programmes: The Impact of BOTA's Conditional Cash Transfer (CCT) Programme 2011-2012
| Name | Country code |
|---|---|
| Kazakhstan | KAZ |
Other Household Survey [hh/oth]
Objectives of the project
The data files described in this documentation correspond to a household sample survey carried out in two rounds (baseline in 2011 and follow-up in 2012) with the objective of evaluating the impact of the BOTA Foundation Conditional Cash Transfer (CCT) programme in small and medium-sized rural areas in Almaty oblast, Kazakhstan.
In addition to impact evaluation, this project also had the following objectives:
- Operational evaluation: Analysis and recommendations on the way in which the programme is being implemented.
- Targeting analysis: Assessment of how effectively the programme's targeting process is reaching the households it is intending to support.
What is the BOTA CCT?
Operational from 2009, this programme offers a regular monthly income to poor households with the aim of improving the lives of children by increasing access to education and other social sector services.
To be eligible for the programme, a household must contain a member that matches one of the four following categories of beneficiary:
- Children aged four and over up until they are eligible to start school (classified by BOTA as the 'Early Childhood Development' (ECD) category);
- Pregnant women, or women with infants up to the age of six months ('Pregnant and Lactating Women' (PLW) category);
- Children with disabilities ('Home-Based Care', HBC);
- Young people aged 16-19 who have completed school and are starting work ('Livelihoods' category)
In addition to meeting the criteria relevant to the beneficiary groups mentioned, possessing the right documentation and living in the area where BOTA is operating, the household must also be classified as 'poor' in order to enrol in the programme. In the application process this is assessed by way of a short computer test administered by BOTA representatives called the 'Proxy Means Test', which aims to maximise inclusion and minimise exclusion of the poorest 30% of the population. Once enrolled, beneficiaries receive the cash for the permitted period of time relevant to their category, provided that they meet specified conditions such as attendance at pre-school or at training sessions.
Sample survey data [ssd]
For the purposes of this study, the principle units of analysis considered are both eligible children of pre-school age, and the households to which they belong.
As described above, there are four categories of beneficiary in the CCT programme. The survey focused on collecting data from a sample of households eligible for the ECD benefit because:
-
these make up the largest proportion of CCT recipients (60% in all areas where BOTA has been operating, as of December 2012)
-
it is possible to use publicly available information to obtain a statistically representative sample of this group; lists of all local children with addresses and dates of birth are compiled twice a year by local government offices in order to ensure that children are attending school; these lists formed the basis of the listing operation described later on.
In order to evaluate the impact of the programme in the context of the ECD category, it was necessary to assess whether there had been any changes to enrolment and attendance at pre-school, attitudes towards pre-school education, households' consumption patterns, and children's home environment (for example, care arrangements, learning materials, etc.).
Where possible the survey team gathered information on the other categories of interest to BOTA; pregnant and lactating women, and children with disabilities - if such people were found in the same households as the children eligible for the ECD benefit. The findings on these categories would not be statistically representative of Almaty oblast but they could nonetheless provide useful insights into attitudes and practices on social and health issues for these groups. The 'Livelihoods' category was not yet one of BOTA's target categories at the time the survey was designed.
The survey team also identified and interviewed where possible, pre-school facilities attended by eligible children; information on pre-school facilities could reveal whether changes on the supply side of pre-school education were likely to have influenced changes observed among households. Results from these interviews would not be statistically representative of Almaty oblast as it was not possible to obtain a sampling frame with all pre-school facilities.
Version
V2.2: Edited, anonymous datasets for public distribution
2015-01-22
In version v2.1, item level data files were separate from main data files. Following suggestions by the WB, item-level data files were restructured and merged with main data files (with the exception of Section G - Household Consumption).
Scope
Themes covered by the survey are as follows:
- Children of pre-school age: Enrolment and attendance at pre-school facilities, care arrangements, early childhood development, child health
- Households: Household characteristics, consumption expenditure, labour supply and sources of income, food security, maternal health, CCT operations
- Individuals: Marital status, citizenship, nationality, parental survivorship, attendance of pre-school and other education, highest level of education achieved, health, employment
- Pre-school facilities: Status of facility, enrolment and attendance, staffing and facilities, the school day, CCT operations
| Topic | Vocabulary |
|---|---|
| Social Protection (includes Pensions, Safety Nets, Social Funds) | World Bank |
Coverage
The survey data are representative of small/medium sized rural areas in Almaty oblast (an oblast is the highest level of territorial unit in Kazakhstan).
Almaty oblast was chosen for the survey because at the time of the baseline survey in 2011, the programme had only just been introduced in some parts. In other oblasts (Akmola and Kyzylorda) it had been operational for over a year. Since 2012, it expanded to Zhambyl, Mangistau and Atyrau oblasts.
Although eligibility for the ECD benefit of the CCT programme is partly determined at the household level, the population of interest are children of pre-school age who were eligible for the ECD benefit at the time of the baseline round of the survey.
To be precise, the children had to be aged 4-6 years, but also had to be eligible for the programme for the whole year between the baseline and follow-up survey.
Producers and sponsors
| Name |
|---|
| Oxford Policy Management Ltd |
| Name | Role |
|---|---|
| Institute for Fiscal Studies, UK | Provided technical inputs into survey design and data analysis |
| BISAM Central Asia, Kazakhstan | Survey implementation team, data processing |
| Name |
|---|
| BOTA Foundation, Kazakhstan |
Sampling
A cluster randomised control trial using multi-stage cluster sampling with no stratification was designed to assess the impact of the CCT.
The intention was to compare characteristics of non-recipient children and their households who would in principle be eligible for the CCT (control group) with those of eligible children and their households residing in communities that had been receiving the CCT for a year (treatment group). These two groups were chosen to be as similar as possible to one another; the baseline round was conducted before the CCT was disbursed served to check that the two groups were the same, and to estimate the size of any differences if these occurred by chance.
The primary sampling unit is the okrug; an okrug is the smallest level of local government administration, consisting of a group of villages governed by a mayor.
Randomisation was conducted at the level of the okrug rather than at the level of the individual for both practical and ethical reasons; BOTA moves from one community to another to conduct enrolment - it cannot reach all communities simultaneously. So at the outset, the randomisation simply defined the order of enrolment by delaying enrolment of the control okrugs until after the enrolment of all the treatment okrugs.
The process of selecting how many children/households to interview, and which ones, consisted of five broad steps:
(1) The selection of all predominantly rural rayons, from a total of 19 rayons in Almaty oblast
A rayon is a subdivision of an oblast; the whole oblast is divided into non-overlapping rayons. Large towns with a population of at least 50,000 that are considered to be major economic or cultural centres ("towns of significance to the oblast") have a status equivalent to a rayon.
"Towns of significance to the oblast" were excluded (3 in total), and the remaining 16 predominantly rural rayons were selected with certainty.
Although BOTA is committed to enrolling eligible households in all areas, initially limiting the rollout to small and medium-sized rural areas allowed BOTA to take advantage of programme volunteers' familiarity with their local community, making it easier to raise awareness of the programme and to encourage potentially eligible households to apply for enrolment in the programme.
(2) The selection of a sample of small/medium sized okrugs, from a total of 254 okrugs
(within the 16 selected rayons)
An okrug is a subdivision of a rayon; the whole rayon is divided into non-overlapping okrugs. Towns that have at least 10,000 people, of whom at least two-thirds are in households where a member is in formal sector employment and where there is some industry, infrastructure, trade and social services ("towns of significance to the rayon") have a status equivalent to an okrug. Some okrugs were excluded according to the following criteria:
- "Towns of significance to the rayon" were excluded (7 in total)
- Rural okrugs with a population of more than 15,000 or at least one settlement with a population of at least 10,000 were excluded (21 in total)
- The remaining 226 okrugs were matched in 113 pairs according to a multidimensional measure of distance based on socio-economic characteristics with each pair composed of two okrugs, the most similar on the basis of available information.
- Three further pairs of okrugs were excluded as BOTA had already launched the CCT programme in at least one okrug in each pair, leaving a universe of 110 pairs (220 okrugs).
- A random sample of 60 pairs was drawn from the universe using the method of 'Probability Proportional to Size' (PPS) with the size of each pair given by the sum of the population of the two okrugs within the pair. 6 pairs of okrugs with large populations were randomly selected twice, resulting in a final number of 54 pairs and representing 108 okrugs.
(3) The random allocation of okrugs to either treatment or control status
For each pair one okrug was randomly assigned to treatment and the other to control. In treatment okrugs the CCT programme began payments immediately after the evaluation baseline survey was conducted. In control okrugs the CCT programme did not operate for the duration of the evaluation.
(4) A listing operation to construct a sample framework for child/household selection
On the basis of lists available from each okrug local government office (both treatment and control okrugs), children of the relevant age group were randomly selected and their households visited; up to 72 children were visited in each okrug (144 children in okrugs that were sampled twice). Eligibility for the programme was determined via the Proxy Means Test and once all data were collected, PMT scores were calculated and a list of eligible children was produced for every okrug. In total, the households of 6,899 children were interviewed, among which 5,388 children were eligible for the CCT.
(5) Selection of eligible children
20 eligible children per okrug (40 children in okrugs that were sampled twice) were randomly selected, thus the intended overall sample was 2,400 children.
It was planned that half of the sample per okrug (1,200 children in total) would be visited for interview at the baseline round and the full sample would be visited for interview at the follow-up round.
The sample design was intended to give an overall household sample size of 1,200 at baseline, and 2,400 at follow-up.
The final sample at the baseline round consisted of 1,173 interviewed children; they were in 1,165 households because in eight households, by chance, two randomly selected children were in the same household. For the follow-up round, only one sampled child per household was interviewed, and the final sample size was 2,289 households.
For households that did not achieve a completed interview, where available a replacement was selected from a random list which had been drawn from the same okrug in order to maintain the required sample size.
At baseline the replacement rate for household interviews was 7%. At follow-up the replacement rate was 13%.
The top-five reasons for replacement interviews at the baseline and follow-up rounds of the survey are as follows:
Baseline: 55% away for extended period, 19% moved away, 15% refused, 7% away for short period, 2% other.
Follow-up: 31% away for extended period, 22% refused, 20% child moved to different household, 17% moved away, 5% other.
For analysis of the survey data, two sets of weights were constructed; child-level weights and household-level weights. These two weights are different because some households have more than one child of eligible age. Sample weights are given as the inverse probability of being selected, calculated by multiplying the probabilities at each sampling stage.
Baseline total population: ch=16,556 ; hh= 16,096
Follow-up total population: ch= 15,971; hh=15,702
At the baseline round, it was not possible to adjust the weights for non-response because insufficient information was retained on the reasons for non-response.
At the follow-up round, both child and household weights were adjusted for non-response, with the treatment sample disaggregated between BOTA beneficiaries and non-beneficiaries.
It was not possible to calculate weights for facilities; therefore, results obtained from these interviews are not statistically representative either of all pre-school facilities (since it was not possible to obtain a sampling frame with all pre-school facilities) or of all pre-school facilities attended by eligible children (since there was a high rate of non-response as many facilities are closed for three months during the summer).
Key analysis variables in the data files
Firstly, the survey round of the data file is indicated by 'bl' (baseline) or 'fu' (follow-up) in the data file name.
The weight variables in the data files are population weights, named 'wt_ch' and 'wt_hh' for child and household weights respectively.
The variable in the data files identifying the Primary Sampling Unit (or okrug) is called 'cluster' and has been anonymised, and the variable identifying whether the okrug (cluster) is treatment or control is called 'areatype'.
The baseline data files include the variable 'bota_ever_ben_MIS', indicating whether the child belongs to a household which was ever a BOTA beneficiary according to BOTA's Management Information System (MIS). The variable was constructed by matching data from BOTA's MIS to the survey data.
The follow-up data files include the variable 'bota_ever_ben', indicating whether the child belongs to a household which was ever a BOTA beneficiary according to BOTA's Management Information System (MIS) OR according to the household (questions qk04, qk14 and qk15). The variable was constructed by matching data from BOTA's MIS to the survey data and combining with information given by the household.
Survey instrument
The survey consisted of a household listing instrument, a household questionnaire for eligible households, and a facility questionnaire for every pre-school facility attended by a sampled child in an eligible household. The questionnaires were developed in English, translated into Russian, pre-tested and piloted before being finalised. For one or two questions, there was translation difficulty as some Russian words do not have an equivalent in Kazakh; this was addressed by providing updated training to interviewers.
The household listing instrument, which served to identify eligible households for the baseline/follow-up survey, included basic information on the household and questions relating to the Proxy Means Test, as well as a small number of additional questions on pre-school enrolment. It was expected to take about the same time to administer as BOTA's application process.
The baseline/follow-up household questionnaires drew on existing standardised household questionnaires that have been implemented in Kazakhstan. They were designed to enable analysis of potential direct and indirect impacts of the programme, including but not limited to: changes in consumption; changes in time use by children and other household members; changes in informal transfers; changes in care-arrangements; attitudes towards pre-school education; enrolment/attendance of pre-school. Questions on issues relating to the operation of the BOTA programme were also included. Questions from sections B (pre-school education), C (care arrangements), D (ECD environment) and E (child health) were asked for sampled children only, with the exception of questions qb45 to qb48 which relate to the respondent's general attitudes towards pre-school education, and qe30 to qe35 which relate to children aged 0-16 at the household level who have a certificate of disability.
The facility questionnaire included questions relating to the supply of pre-school education, such as the size of the facility, number of staff, basic equipment and quality. The fieldwork team sought to talk directly with the school director or a member of management.
Data collection
| Start | End | Cycle |
|---|---|---|
| 2011-04 | 2011-12 | Listing |
| 2011-06 | 2011-12 | Baseline |
| 2012-06 | 2012-12 | Follow-up |
| Name |
|---|
| BISAM Central Asia, Kazakhstan |
Interviewer training
The interviewers were trained in both general interviewing techniques and survey specific issues; the general training consisted of preparation for being in the field, such as discussions on what to expect in the field (weather, having water, food, transport, permissions), making introductions and how to handle situations when people did not want to be interviewed.
The survey specific training involved an introduction to the entire survey, going through each and every question, and practising filling out forms.
For all survey rounds, both the general and survey specific training took around three days. In addition, there was a one-day pilot in field (visiting households that were not part of the sample), and one day of discussing ethnical/practical issues.
Supervision
As well as interviewers reviewing their own questionnaires, the team leaders in the field (in charge of about three interviewers) also reviewed the questionnaires. After the team leader had collected all the questionnaires, they were given to a supervisor who reviewed a selection of them. There were two supervisors, one to cover the teams in the northern half of Almaty oblast and the other for the southern half. In the event of any problems, either the supervisor or fieldwork manager would go back to the field with the teams to resolve any issues.
Fieldwork teams conducted a listing operation in each okrug from which eligible households were selected for the survey between April and December 2011.
The baseline round of the survey was carried out in June to December 2011, before households started receiving the CCT. Household interviews were conducted as soon as possible after listing operations were completed.
The follow-up round was carried out in June to December 2012, one year later to reduce the impact of seasonality. The one-year period between surveys was selected because most households in the programme receive the transfer for only a short time and many would not still be enrolled after two years.
On average, the listing questionnaire took about 10-15 minutes to administer; the household questionnaire took about 1 hour 15 minutes; and the facility questionnaire took about 50 minutes. Interviews were conducted in Russian or Kazakh as appropriate, requiring advance preparation of questionnaire copies in both languages and a fieldwork team with mixed language skills.
Regarding location, in the villages around Almaty and Taldy-Korgan it was sometimes hard to find parents at home since some of them work in the city. This bias in non-response was taken into account when adjusting the sampling weights.
Data processing
The following procedures were implemented in order to control for data consistency and coherence:
- SPSS syntax used to check data files for each okrug following double data-entry
The syntax generated reports detailing the inconsistencies between the double-entered data files, and allowed the user to perform as many rounds of checks/corrections as required. The reports were used to manually check and correct inconsistencies by referring to the information recorded on the paper questionnaires.
- SPSS syntax used for skip and range checks
The syntax verified that the data followed the correct skip patterns and that categorical variables adhered to the correct value ranges as specified in the questionnaire. As above, it generated reports which were used to manually check and correct the data files by referring to the corresponding paper questionnaires.
- Manual correction of some household member positions
Following review of the household member data files (mainly by comparing names and dates of birth between the baseline and follow-up rounds), some household member positions were manually corrected in the follow-up household roster (section A) for households that were re-interviewed after the baseline round in order to enable direct comparison among household members between survey rounds.
- Any other data imputation (such as treatment of outliers) was performed during the analysis phase, with edits not captured in the final anonymised data files.
Data appraisal
Statistically significant differences between treatment and control locations, or between beneficiaries and non-beneficiaries in treatment areas, are presented in report tables* with a series of asterisks, indicating significance at the 10%, 5% and 1% levels. All statistical analyses were performed with Stata, using its 'svy' facilities for survey data analysis to account for the sampling design.
- Please refer to reports produced by Oxford Policy Management Ltd
Depositor information
| Name |
|---|
| Oxford Policy Management Ltd |
Data Access
| Name | URL | |
|---|---|---|
| Oxford Policy Management Ltd | http://www.opml.co.uk/ | admin@opml.co.uk |
The data files have been anonymised and are available as a Public Use Dataset. They are accessible to all for statistical and research purposes only, under the following terms and conditions:
- The data and other materials will not be redistributed or sold to other individuals, institutions, or organisations without the written agreement of Oxford Policy Management Ltd.
- The data will be used for statistical and scientific research purposes only. They will be used solely for reporting of aggregated information, and not for investigation of specific individuals or organisations.
- No attempt will be made to re-identify respondents, and no use will be made of the identity of any person or establishment discovered inadvertently. Any such discovery would immediately be reported to Oxford Policy Management Ltd.
- No attempt will be made to produce links among datasets provided by Oxford Policy Management Ltd, or among data from Oxford Policy Management Ltd and other datasets that could identify individuals or organisations.
- Any books, articles, conference papers, theses, dissertations, reports, or other publications that employ data obtained from Oxford Policy Management Ltd will cite the source of data in accordance with the Citation Requirement provided with each dataset.
- An electronic copy of all reports and publications based on the requested data will be sent to Oxford Policy Management Ltd.
The original collector of the data, Oxford Policy Management Ltd, and the relevant funding agencies bear no responsibility for use of the data or for interpretations or inferences based upon such uses.
Oxford Policy Management Ltd, External Evaluation of the Kazakhstan BOTA CCT Programme 2011-2012, Version 2.2 of the public use dataset (January 2015)
Disclaimer and copyrights
The user of the data acknowledges that the original collector of the data, the authorised distributor of the data, and the relevant funding agency bear no responsibility for use of the data or for interpretations or inferences based upon such uses.
(c) 2015, Oxford Policy Management Ltd
Contacts
| Name | Affiliation | URL | |
|---|---|---|---|
| Clare O'Brien, Project Manager | Oxford Policy Management Ltd | clare.obrien@opml.co.uk | http://www.opml.co.uk/people-partners/consultants/clare-obrien |
| Stephi Springham, Data Analyst | Oxford Policy Management Ltd | stephi.springham@opml.co.uk | http://www.opml.co.uk/people-partners/consultants/stephi-springham |
| Oxford Policy Management Ltd | http://www.opml.co.uk/projects/external-evaluation-bota-foundations-social-sector-programmes-kazakhstan |
Metadata production
DDI_KAZ_2011_BOTACCT_v01_M
| Name | Affiliation | Role |
|---|---|---|
| Stephi Springham | Oxford Policy Management Ltd | Data Analyst |
2015-01-23
Metadata version
Version 01 (January 2015). First version of study documentation for public distribution.
Version 02 (January 2015). Revised first version of study documentation for public distribution, following suggestions by the WB; Item-level data files restructured and merged with main data files (with the exception of Section G - Household Consumption), Access Authority field completed, literal question fields completed.
Version 03 (January 2015). Edited version based on Version 01 DDI (DDI-OPM-6838-KAZ-V1-1-2015) that was done by Oxford Policy Management Ltd.