Survey ID Number
TZA_2018_EQUIPIE-EL_v01_M
Title
Education Quality Improvement Programme Impact Evaluation Endline Survey 2018
Notes
The scope of the EQUIP-T IE Endline Survey includes:
-HEAD TEACHER/HEAD COUNT/SCHOOL RECORDS: Head teacher background information, frequency/type of school leadership and management in-service training received, availability and contents of school development plan, teacher management, frequency of staff meetings, ward education officer supervision and support to the school, JUU club activities, school committee, school information system, Parent-Teacher-Partnership activities, community engagement, head teacher morale and other conditions of service, head teacher attendance, reasons for head teacher and teacher absenteeism (reported by head teachers), teacher attendance (from school records and by headcount on the day of the survey), teacher punctuality, pupil attendance (from school records and by headcount on the day of the survey), pupil enrolment, school background information (teachers, physical facilities, school timetable, number of days school open), school characteristics, infrastructure and funding, receipt of in-kind resources.
-STANDARD 3 PUPILS: Pupil background information, pupils' school experience, Kiswahili Early Grade Reading Assessment (EGRA) and Early Grade Mathematics Assessment (EGMA) based on standards 1 and 2 national curriculum requirements. Note: The same pupils were assessed in both Kiswahili and mathematics.
-PARENTS OF SAMPLED STANDARD 3 PUPILS: household and parental characteristics (that can be used to convert scores into poverty likelihoods based on a pre-existing instrument), pupil background, home support for schooling, parent-school communication.
-TEACHERS WHO TEACH STANDARDS 1-3 KISWAHILI AND/OR MATHEMATICS: Interview including background information, qualifications, frequency/type of in-service training received, classroom teaching and pupil assessment practices, access to teaching and learning resources, support for teaching (lesson planning, observation, meetings), frequency/nature of performance appraisal and teacher morale and other conditions of service.
-LESSON OBSERVATION: Standard 2 Kiswahili and mathematics lesson observations of inclusive behaviour of teachers with respect to gender and spatial location of pupils, key teaching practices in the classroom, pupils' reading and teacher support, availability of lesson plan, availability of seating, availability and use of teaching and learning materials during the lesson.
-TEACHER GROUP INTERVIEW (including In-service Training Coordinator): Frequency and nature of all EQUIP-T early grade teacher training sessions that have taken place away from school and in school since baseline, INCO background information, participation in ward cluster reflection meetings.
Sampling Procedure
Because the EQUIP-T regions and districts were purposively selected (see 'EQUIP-Tanzania Impact Evaluation. Final Baseline Technical Report, Volume I: Results and Discussion' under Reports and policy notes), the IE sampling strategy used propensity score matching (PSM) to: (i) match eligible control districts to the pre-selected and eligible EQUIP-T districts (see below), and (ii) match schools from the control districts to a sample of randomly selected treatment schools in the treatment districts. The same schools are surveyed for each round of the IE (panel of schools) and a cross section of standard 3 pupils and Standard 1-3 teachers will be interviewed at each round of the survey (no pupil panel or teacher panel).
------------------------------------------------
Identifying districts eligible for matching
------------------------------------------------
Eligible control and treatment districts were those not participating in any other education programme or project that may confound the measurement of EQUIP-T impact. To generate the list of eligible control and treatment districts, all districts that are contaminated because of other education programmes or projects or may be affected by programme spill-over were excluded as follows:
-All districts located in Lindi and Mara regions as these are part of the EQUIP-T programme but implementation started later in these two regions (the IE does not cover these two regions);
-Districts that will receive partial EQUIP-T programme treatment or will be subject to potential EQUIP-T programme spillovers;
-Districts that are receiving other education programmes/projects that aim to influence the same outcomes as the EQUIP-T programme and would confound measurement of EQUIP-T impact;
-Districts that were part of pre-test 1 (two districts); and
-Districts that were part of pre-test 2 (one district).
-------------------
Sampling frame
-------------------
To be able to select an appropriate sample of pupils and teachers within schools and districts, the sampling frame consisted of information at three levels:
-District;
-School; and
-Within school.
The sampling frame data at the district and school levels was compiled from the following sources: the 2002 and 2012 Tanzania Population Censuses, Education Management Information System (EMIS) data from the Ministry of Education and Vocational Training (MoEVT) and the Prime Minister's Office for Regional and Local Government (PMO-RALG), and the UWEZO 2011 student learning assessment survey. For within school level sampling, the frames were constructed upon arrival at the selected schools and was used to sample pupils and teachers on the day of the school visit.
-------------------
Sampling stages
-------------------
Stage 1: Selection of control districts
--------------------------------------------
Because the treatment districts were known, the first step was to find sufficiently similar control districts that could serve as the counterfactual. PSM was used to match eligible control districts to the pre-selected, eligible treatment districts using the following matching variables: Population density, proportion of male headed households, household size, number of children per household, proportion of households that speak an ethnic language at home, and district level averages for household assets, infrastructure, education spending, parental education, school remoteness, pupil learning levels and pupil drop out.
Stage 2: Selection of treatment schools
-------------------------------------------------
In the second stage, schools in the treatment districts were selected using stratified systematic random sampling. The schools were selected using a probability proportional to size approach, where the measure of school size was the standard two enrolment of pupils. This means that schools with more pupils had a higher probability of being selected into the sample. To obtain a representative sample of programme treatment schools, the sample was implicitly stratified along four dimensions:
-District;
-PSLE scores for Kiswahili;
-PSLE scores for mathematics; and
-Total number of teachers per school.
Stage 3: Selection of control schools
----------------------------------------------
As in stage one, a non-random PSM approach was used to match eligible control schools to the sample of treatment schools. The matching variables were similar to the ones used as stratification criteria: Standard two enrolment, PSLE scores for Kiswahili and mathematics, and the total number of teachers per school.
The endline survey was conducted for the same schools as the baseline and midline surveys (a panel of schools). However, the IE does not have a panel of pupils or teachers as a pupil only attends standard three once (unless repeating) and there is high teacher turnover. Thus, the IE sample is a repeated cross-section of pupils and teachers in a panel of schools.
Stage 4: Selection of pupils and teachers within schools
-------------------------------------------------------------------
Pupils were sampled within schools using systematic random sampling based on school registers. The within-school sampling was assisted by selection tables automatically generated within the computer assisted survey instruments. Per school, 15 standard 3 pupils were sampled. The parents of these 15 sampled pupils were then interviewed using the poverty scorecard instrument.
For the teacher interviews, as at midline, all teachers of Standards 1-3 who teach Kiswhaili or maths were interviewed to boost the sample size as many schools are small (as opposed to baseline where up to three teachers were sampled within each school for the interviews).
Lesson observations were not randomly sampled. Instead, one maths and one Kiswahili Standard 2 lessons were selected within each school using convenience sampling to be observed on the day of the survey.
-------------------------
Replacement sample
-------------------------
At baseline, if a selected school could not be surveyed it was replaced. In the process of sampling, the impact evaluation team drew a replacement sample of schools, which was used for this purpose (reserve list) and the use of this list was carefully controlled. Five out of the 200 original baseline sample schools were replaced during the fieldwork. At midline and endline, all of the 200 schools surveyed at baseline were visited again (no replacements).
---------------
Sample sizes
---------------
The actual sample sizes at endline are:
-200 schools (100 treatment and 100 control).
-2,999 standard 3 pupils assessed in both Kiswahili and mathematics.
-2,992 poverty scorecards were administered to the assessed pupils' parent(s).
-889 teachers who teach standards 1 to 3 Kiswahili and/or mathematics interviewed.
-196 standard 2 Kiswahili and mathematics lessons observed (treatment schools only).
-99 teacher group interviews were conducted (treatment schools only).
Note that the lesson observation and the small group teacher interview were only conducted in treatment schools, because the information generated could not be used in the impact modelling and so collecting information in control schools was not necessary.
-------------------------
Representativeness
-------------------------
The results from the treatment schools are representative of government primary schools in the 17 EQUIP-T programme treatment districts. However, the results from the schools in the 8 control districts are NOT representative because these districts were not randomly sampled but matched to the 17 treatment districts using propensity score matching.
Data Collection Notes
------------
Personnel
------------
Oxford Policy Management's (OPM) Tanzania office conducted the endline IE survey.
The fieldwork management team comprised eight members (including six OPM staff) led by a quantitative survey project manager who had overall responsibility for the design, implementation, management and quality of the fieldwork. Since all the survey instruments were administered using computer assisted personal interviewing (CAPI), the team also included several members with strong computer programming skills in the relevant software (Surveybe). The overall project manager for the IE, who is responsible for the content of the instruments worked closely with the fieldwork team during pre-testing, training, piloting and early fieldwork. 60 enumerators were invited to the training. These were selected based on the following criteria (in order): (i) high performance during the EQUIP-T baseline and midline surveys (24 enumerators from BL and/or ML attended the EL training); (ii) interviewers with strong track record from other OPM-led surveys; and (iii) new recruits that were selected based on their prior survey experience and knowledge of education.
--------------------------
Fieldwork preparation
--------------------------
The early fieldwork preparation consisted of pre-testing the instruments and protocols and refining the instruments and protocols, obtaining permits from the government for visiting schools during the pre-tests, training and pilot and fieldwork, and revising the ML fieldwork manual.
Pre-tests of instruments
-----------------------------
A full pre-test of all instruments and protocols took place from 19 to 23 February 2018 in Dodoma. A team of six (four members of the core evaluation team and two experienced survey supervisors who were supervisors during the midline fieldwork) visited eight schools, following one day of classroom based training. The main objectives of the pre-test were to test the functionality of the updated electronic questionnaires in the latest version of the CAPI software (Surveybe); test the changes that were made to the midline instruments, focusing mostly on the head teacher interview; and test the new endline instrument - that is the teacher group interview. The pre-test resulted in the following outcomes:
-Refinement of the instruments and data collection protocols;
-Refinement of the translation of instruments from English to Kiswahili; and
-Significant changes made to the development of the instruments in CAPI (Surveybe).
Permits and reporting
--------------------------
As part of preliminary preparations for any survey in Tanzania, there are two types of governmental permits that have to be obtained prior the beginning of Research work:
-COSTECH Permit - Mandatory for any research activity in Tanzania.
-Ministry Permit - Different partners in the field require Ministry letters, as few recognise COSTECH. These permits give the order to local administration to cooperate with the research and support the field teams.
Upon receipt of the permits, the anticipated fieldwork needs to be reported at the regional and district level. Letters introducing the study to local leaders are obtained in the process. For the endline survey, the COSTECH research clearance and an introduction letter were received more than three months prior to the start of actual fieldwork. For the Ministry permits, OPM reported to The Prime Minister’s Office Regional Administration and Local government (PMORALG) and to the Ministry of Education and Vocational Training (MOEVT). Reporting to MOEVT was relatively fast and simple. The initial submitted letters were followed up in person, and an introduction letter to all 12 Regional Administrative Secretaries (RAS) was received after seven days. Getting government approvals from PMORALG and the RASs was more challenging and time-consuming as it required physical reporting to PMORALG’s office in Dodoma as well as physical reporting to all regions and districts that are covered by the endline fieldwork, pre-testing and piloting. However, having learned from midline how challenging this process is, the fieldwork management team devised a plan for endline that started the reporting process early on and involved two members of the fieldwork management team and two supervisors physically reporting in person to all 12 regional and 25 district offices during the month of February. This resulted in all permits and approval letters being obtained at least one month prior to piloting.
Fieldwork manual
---------------------
Using the midline fieldwork manual as a basis, an extensive endline fieldworker manual was developed that covered basic guidelines on behaviour and attitude, the use of CAPI and data validation procedures, instructions on fieldwork plans and procedures (sample, targets, replacements, communication, and reporting) as well as a dedicated part on the description of all instruments and protocols. Insights from the pre-test were reflected in the manual. Draft versions of the instrument and protocol sections of the manuals were printed, handed out to interviewers as a reference during the training, and used as guidelines by the trainers. The manual was updated on an ongoing basis during the training and pilot phase where updated conventions or additional clarifications were needed. The final version of the manual was printed at the end of the pilot phase and copies provided to the field teams.
Training and pilot
---------------------
Enumerator training and a field pilot took place in Dar es Salaam and Dodoma from 26 March to 14 April 2018. A total of 60 enumerator trainees participated in the training. The training was delivered by four members of the fieldwork management team, the overall IE project manager and another member of the core evaluation team. The main objective of the training was to ensure that team members would be able to master the instruments, understand and correctly implement the fieldwork protocols, comfortably use CAPI, and be able to perform data validation. Supervisors were furthermore trained on their extra responsibilities of data management, fieldwork and financial management, logistical tasks, and the transmission of data files to the data manager.
The training had two components: a classroom-based training component and a field-based component that included a full scale pilot. The performance of enumerators was assessed on an on-going basis, using written assessments and observation of performance in the field and these scores were recorded. At the end of the training and pilot phase, the final fieldwork team was selected using this information.
A higher number of data collectors than needed for data collection were invited to and attended the training. This allowed for a selection of the best suited candidates at the end of the training and provided a pool of reserve additional trained staff that could be called upon in case of enumerator attrition during data collection.
Fieldwork organisation
---------------------------
The fieldwork plan was designed to cover all 200 schools within all 12 regions and 25 districts for the duration of not more than six weeks. The plan had to cater for the short fieldwork time window dictated by the end of the school mid-term break and the start of exams at the end of the term; rainy season; allowing the fieldwork management team to supervise teams during the first week of implementation; minimising travel days between districts and during the weekdays; suitable allocation of teams to districts to address cultural and language barriers; and flexibility to deal with unforeseen circumstances.
The team composition and fieldwork model at endline were the same as those at midline with the exception of adding one more field team to deal with the shorter timeframe at endline and to ensure that the fieldwork is completed within five to six weeks. At endline there were four treatment teams composed of five enumerators and one supervisor, four control teams of four enumerators and one supervisor each, and one team of five enumerators and one supervisor that visited control and treatment areas. Each team visited and completed one school on one day.
The fieldwork started on 16 April and ended on 21 May 2018 with no major breaks in-between.