Survey ID Number
TZA_2016_EQUIPIE-ML_v02_M
Title
Education Quality Improvement Programme Impact Evaluation Midline Survey 2016
Sampling Procedure
Because the EQUIP-T regions and districts were purposively selected (see 'EQUIP-Tanzania Impact Evaluation. Final Baseline Technical Report, Volume I: Results and Discussion' under Reports and policy notes), the IE sampling strategy used propensity score matching (PSM) to: (i) match eligible control districts to the pre-selected and eligible EQUIP-T districts (see below), and (ii) match schools from the control districts to a sample of randomly sampled treatment schools in the treatment districts. The same schools are surveyed for each round of the IE (panel of schools) and standard 3 pupils will be interviewed at each round of the survey (no pupil panel).
Identifying districts eligible for matching
Eligible control and treatment districts were those not participating in any other education programme or project that may confound the measurement of EQUIP-T impact. To generate the list of eligible control and treatment districts, all districts that are contaminated because of other education programmes or projects or may be affected by programme spill-over were excluded as follows:
- All districts located in Lindi and Mara regions as these are part of the EQUIP-T programme but implementation started later in these two regions (the IE does not cover these two regions);
- Districts that will receive partial EQUIP-T programme treatment or will be subject to potential EQUIP-T programme spillovers;
- Districts that are receiving other education programmes/projects that aim to influence the same outcomes as the EQUIP-T programme and would confound measurement of EQUIP-T impact;
- Districts that were part of pre-test 1 (two districts); and
- Districts that were part of pre-test 2 (one district).
Sampling frame
To be able to select an appropriate sample of pupils and teachers within schools and districts, the sampling frame consisted of information at three levels:
- District;
- School; and
- Within school.
The sampling frame data at the district and school levels was compiled from the following sources: the 2002 and 2012 Tanzania Population Censuses, Education Management Information System (EMIS) data from the Ministry of Education and Vocational Training (MoEVT) and the Prime Minister's Office for Regional and Local Government (PMO-RALG), and the UWEZO 2011 student learning assessment survey. For within school level sampling, the frames were constructed upon arrival at the selected schools and was used to sample pupils and teachers on the day of the school visit.
Sampling stages
Stage 1: Selection of control districts
Because the treatment districts were known, the first step was to find sufficiently similar control districts that could serve as the counterfactual. PSM was used to match eligible control districts to the pre-selected, eligible treatment districts using the following matching variables: Population density, proportion of male headed households, household size, number of children per household, proportion of households that speak an ethnic language at home, and district level averages for household assets, infrastructure, education spending, parental education, school remoteness, pupil learning levels and pupil drop out.
Stage 2: Selection of treatment schools
In the second stage, schools in the treatment districts were selected using stratified systematic random sampling. The schools were selected using a probability proportional to size approach, where the measure of school size was the standard two enrolment of pupils. This means that schools with more pupils had a higher probability of being selected into the sample. To obtain a representative sample of programme treatment schools, the sample was implicitly stratified along four dimensions:
- District;
- PSLE scores for Kiswahili;
- PSLE scores for mathematics; and
- Total number of teachers per school.
Stage 3: Selection of control schools
As in stage one, a non-random PSM approach was used to match eligible control schools to the sample of treatment schools. The matching variables were similar to the ones used as stratification criteria: Standard two enrolment, PSLE scores for Kiswahili and mathematics, and the total number of teachers per school.
The midline survey was conducted for the same schools as the baseline survey (a panel of schools) and the endline survey in 2018 will cover the same sample of schools. However, the IE does not have a panel of pupils as a pupil only attends standard three once (unless repeating). Thus, the IE sample is a repeated cross-section of pupils in a panel of schools.
Stage 4: Selection of pupils and teachers within schools
Pupils and teachers were sampled within schools using systematic random sampling based on school registers. The within-school sampling was assisted by selection tables automatically generated within the computer assisted survey instruments.
Per school, 15 standard 3 pupils were sampled. For the teacher development needs assessment (TDNA), in the sample treatment schools, up to three teachers of standards 1 to 3 Kiswahili, up to three teachers of standards 1 to 3 mathematics; and up to three teachers of Standards 4-7 mathematics were randomly sampled. For the teacher interview sampling, one change was made at midline, instead of sampling up to three teachers of Standards 1-3 all of them were interviewed to boost the sample size as many schools are small.
Replacement sample
At baseline, if a selected school could not be surveyed it was replaced. In the process of sampling, the impact evaluation team drew a replacement sample of schools, which was used for this purpose (reserve list) and the use of this list was carefully controlled. Five out of the 200 original baseline sample schools were replaced during the fieldwork. At midline, all of the 200 schools surveyed at baseline were visited again (no replacements).
Sample sizes
The actual sample sizes at midline are:
- 200 schools (100 treatment and 100 control).
- 2,971 standard 3 pupils assessed in both Kiswahili and mathematics.
- 2,963 poverty scorecards were administered to the assessed pupils' parent(s).
- 817 teachers who teach standards 1 to 3 Kiswahili and/or mathematics interviewed
- 243 teachers who teach standards 1 to 3 Kiswahili were administered the Kiswahili teacher development needs assessment (treatment schools only).
- 239 teachers who teach standards 1 to 3 mathematics were administered the mathematics teacher development needs assessment (treatment schools only).
- 231 teachers who teach standards 4-7 mathematics were administered the mathematics teacher development needs assessment (treatment schools only).
- 231 standard 2 Kiswahili and mathematics lessons observed (treatment schools only).
Representativeness
The results from the treatment schools are representative of government primary schools in the 17 EQUIP-T programme treatment districts. However, the results from the schools in the 8 control districts are NOT representative because these districts were not randomly sampled but matched to the 17 treatment districts using propensity score matching (see above).
Data Collection Notes
Personnel: Oxford Policy Management's (OPM) Tanzania office conducted the Midline Impact Evaluation survey.
The fieldwork management team comprised seven members (including six OPM staff) led by a quantitative survey project manager who had overall responsibility for the design, implementation, management and quality of the fieldwork. Since all the survey instruments excluding the teacher development needs assessments (TDNAs) were administered using computer assisted personal interviewing (CAPI), the team also included several members with strong computer programming skills in the relevant software (Surveybe). The overall project manager for the IE, who is responsible for the content of the instruments worked closely with the fieldwork team during pre-testing, training, piloting and early fieldwork. 51 enumerators were invited to the training. These were selected based on the following criteria (in order): (i) high performance during the EQUIP-T BL survey (about half of the enumerators from BL also worked in the ML survey); (ii) interviewers with strong track record from other OPM-led surveys; and (iii) new recruits that were interviewed over 2 days and selected based on their prior survey experience and knowledge of education.
Fieldwork preparation: The early fieldwork preparation consisted of pre-testing the instruments and protocols, obtaining permits from the government for visiting schools during the pre-tests, pilot and fieldwork, revising the BL fieldwork manual, and refining the instruments and protocols.
Pre-tests of instruments: See Questionnaires Section below.
Permits and reporting
As part of preliminary preparations for any survey in Tanzania, there are two types of governmental permits that have to be obtained prior the beginning of Research work:
- COSTECH Permit - Mandatory for any research activity in Tanzania.
- Ministry Permit - Different partners in the field require Ministry letters, as few recognise COSTECH. These permits give the order to local administration to cooperate with the research and support the field teams.
Upon receipt of the permits, the anticipated fieldwork needs to be reported at the regional and district level. Letters introducing the study to local leaders are obtained in the process. For the ML IE survey, the COSTECH research clearance and an introduction letter was received two months prior the start of actual fieldwork. For the Ministry permits, OPM reported to The Prime Minister's Office Regional Administration and Local government (PMORALG) and to the Ministry of Education and Vocational Training (MoEVT). Reporting to MoEVT was relatively fast and simple. The initial submitted letters were followed up in person, and an introduction letter to all 12 Regional Administrative Secretaries was received after 7 days. Getting government approvals from PMORALG proved very time-consuming. The final decision was to shift to a physical reporting approach, as sending letters by courier and follow-up phone calls were unsuccessful. In a combined effort, three of the fieldwork management team members reported in person to all 10 regional and 25 district offices during the enumerator training period. In total 50 person days (including travel days, as distances are vast) had to be allocated to this final reporting task.
Fieldwork manual
Using the baseline fieldwork manual as a basis, an extensive midline fieldworker manual was developed that covered basic guidelines on behaviour and attitude, the use of CAPI and data validation procedures, instructions on fieldwork plans and procedures (sample, targets, replacements, communication, and reporting) as well as a dedicated part on the description of all instruments and protocols. Insights from the pre-test were reflected in the manual. Draft versions of the instrument and protocol sections of the manuals were printed, handed out to interviewers as a reference during the training, and used as guidelines by the trainers. The manual was updated on an ongoing basis during the training and pilot phase where updated conventions or additional clarifications were needed. The final version of the manual was printed at the end of the pilot phase and copies provided to the field teams.
Training and pilot
Enumerator training and a field pilot took place in Dar es Salaam and Dodoma from 29th March to 14th April. A total of 47 enumerator trainees participated in the training. The training was delivered by four members of the fieldwork management team and the overall IE project manager. The main objective of the training was to ensure that team members would be able to master the instruments, understand and correctly implement the fieldwork protocols, comfortably use CAPI, and be able to perform data validation. Supervisors were furthermore trained on their extra responsibilities of data management, fieldwork and financial management, logistical tasks, and the transmission of data files to the data manager.
The training had two components: a classroom-based training component and a field-based component that included a full scale pilot. The performance of enumerators was assessed on an on-going basis, using written assessments and observation of performance in the field and these scores were recorded. At the end of the training and pilot phase, the final fieldwork team was selected using this information.
Fieldwork organisation
The fieldwork plan was designed to cover all 200 schools within all 12 regions and 25 districts for the duration of not more than 7 weeks starting April 15th 2016 to May 27th 2016. Teams communicated regularly with OPM to report delays and/or any event likely to affect the feasibility of the fieldwork plan.
The team composition and fieldwork model at ML were set up differently to baseline to: a) reduce transport costs by reducing car days relative to fieldworker days and moving more travel days to Saturday (schools closed, but working day for fieldworkers), and b) to be able to translate the reduced requirements of instruments in control schools into reduced team size for control teams. At baseline, fieldwork was undertaken by 15 teams of 3 fieldworkers each visiting a school on two consecutive days. At midline, 4 treatment teams of 6 fieldworkers (1 supervisor and 5 enumerators) and 4 control teams of 5 fieldworkers (1 supervisor and 5 enumerators) visited one school on one day. Each team had one supervisor who was responsible for quality-checking the enumerators' work.
The fieldwork started on the 15th of April 2016 and ended on the 27th of May 2016 with no major breaks in-between.