Southern Africa Consortium for Monitoring Educational Quality 2007
The SACMEQ III Project commenced in 2006 and was completed during 2011. The SACMEQ III data collection was implemented in fifteen SACMEQ Ministries of Education (Botswana, Kenya, Lesotho, Mauritius, Malawi, Mozambique, Namibia, Seychelles, South Africa, Swaziland, Tanzania (Mainland), Tanzania (Zanzibar), Uganda, Zambia, and Zimbabwe). The SACMEQ III Project followed the general research direction of the first two SACMEQ Projects by focusing on an examination of the conditions of schooling in relation to achievement levels of learners and their teachers in reading, and mathematics. The focus was expanded to cover the learners’ levels of basic knowledge about HIV and AIDS. The SACMEQ III Project involved data collections from around 61,000 learners, 8,000 teachers, and 2,800 school principals.
Kind of Data
Sample survey data [ssd]
Unit of Analysis
- v2.1: Edited, anonymous dataset for public distribution.
basic skills education [6.1]
The lowest level of geographic aggregation covered by the data is province, and in some cases, metropolitan area.
Unit of Analysis
The desired target population for the SACMEQ III study was defined as "All pupils at Grade 6 level in 2007 (at the first week of the eighth month of the school year) who were attending registered mainstream primary schools". This definition used a grade-based description (and not an age-based description) of pupils because an age-based description would have required the collection of data across many grade levels due to the high incidences of "late starters" and grade repetition in SACMEQ school systems.
Producers and sponsors
Authoring entity/Primary investigators
Southern and Eastern Africa Consortium for Monitoring Educational Quality (SACMEQ)
United Nations Educational, Scientific and Cultural Organization
Funding the project
Funding and Technical Support
Ministries of Education
Funding the project
The desired target population definition for the SACMEQ III Project was exactly the same (except for the year) as was employed for the SACMEQ I and II Projects. This consistency was maintained in order to make valid cross-national and cross-time estimates of "change" in the conditions of schooling and the quality of education.
The SACMEQ III data were selected using a stratified two-stage cluster sample design based on the technique of a lottery method of sampling proportional to size, with the assistance of SAMDEM software (Sylla et al., 2003). At the first stage, schools were selected in each region (province) in proportion to the number of pupils in that region in the defined target population. The main reason for choosing Region as the explicit stratification variable was that the SACMEQ Ministries of Education wanted to have education administration regions as "domains" for the study. That is, the Ministries wanted to have a reasonably accurate sample estimates of population characteristics for each region. At the second stage, a simple random sample of 25 pupils was taken within each selected school (in the Seychelles, all Grade 6 pupils in all 25 schools in the island country were tested).
In educational survey research the primary sampling units that are most often employed (schools) are rarely equal in size. This variation causes difficulties with respect to the control of the total sample size when schools are selected with equal probability at the first stage of a multi-stage sample design. One method of obtaining greater control over the total sample size is to stratify the schools accorging to size and then select samples of schools within each stratum. A more widely applied alternative is to employ probability proportional to size (PPS) sampling of schools within strata followed by the selection of a simple random sample size and results in epsem sampling of pupils within strata. The lottery method of PPS selection was implemented for the SACMEQ Projects with the assistance of the SAMDEM software (Sylla et al, 2003).
In order to avoid selection bias, precautions were taken to ensure that school heads and teachers did not have any influence over the sampling procedures within schools. This is because school heads and teachers might have felt they had a vested interest in selecting particular kinds of pupils, and this could have resulted in major distortions of sample estimates (Brickell, 1974).
The planned Mauritian sample was 152 schools and 3950 pupils. There were six schools with two sample groups due to large number of Standard 6 pupils. The achieved sample comprised 152 schools and 3524 pupils. Pupils who were sampled in the sampled schools but were not available on the day of data collection were not replaced.
Stratification used to define population
In all, seven regions were defined as strata for the target population as follows:
Region 1, Port Louis & North (1PL): government schools in Education Zone 1.
Region 2, Beau Bassin & East (2BB): government schools in Education Zone 2.
Region 3, Curepipe & South (3CU): government schools in Education Zone 3.
Region 4, Vacoas & West (4VA): government schools in Education Zone 4.
Region 5, Rodrigues (5RO): all schools in the Island of Rodrigues.
Region 6, Black River (6BR): all schools in Black River District.
Region 7, Private (7PR): private schools from different education zones.
Deviations from the Sample Design
Excluded Target Population
One of the rules followed by SACMEQ for ensuring valid data in large-scale studies is that no more than 5 percent of the learners in the desired target population may be excluded from the defined target population. Like in SACMEQ II, special schools which provide education to learners with severe educational needs were excluded from the SACMEQ III sample. “Small” mainstream schools which had less than 15 learners enrolled in Grade 6 in 2007 were also allocated to the excluded population to reduce data collection costs – without the risk of leading to major distortions in the study population.
Response rate in surveys refers to the percentage of the total sample units that were planned who actually participate in the study. The SACMEQ rule is that the overall response rate for both the schools and the learners should not be less than 90%. In the SACMEQ III project the Mauritian overall response rates for schools and pupils were 100% and 89%, respectively.
The calculation of sampling weights could only be conducted after all files had been cleaned and merged. Sampling weights were used to adjust for missing data and for variations in probabilities of selection that arose from the application of stratified multi-stage sample designs. There were also certain country-specific aspects of the sampling procedures, and these had to be reflected in the calculation of sampling weights.
Two forms of sampling weights were prepared for the SACMEQ III Project. The first sampling weight (RF2) was the inverse of the probability of selecting a pupil into the sample. These “raising factors” were equal to the number of pupils in the defined target population that were “represented by a single pupil” in the sample. The second sampling weight (pweight2) was obtained by multiplying the raising factors by a constant so that the sum of the sampling weights was equal to the achieved sample size. A detailed account of weighting procedures can be found in Ross et al (2004).
Dates of Data Collection (YYYY/MM/DD)
Mode of data collection
Type of Research Instrument
- Apart from pupil achievement scores, SACMEQ studies are renowned for collecting a wide range of information about pupils, teachers, classrooms, school heads, schools, and school communities. For the SACMEQ III study, four main questionnaires (pupil, teacher, school head, and school information) were used.
- It is important to note that SACMEQ questionnaires were subjected to careful thought, thorough examination, and stringent refinement before they were administered. For example, for the SACMEQ III study, the questionnaires were developed by a committee of experts consisting of members drawn from all SACMEQ countries, SCC staff, IIEP staff, and private consultants, following
(a) Field experiences gained from the SACMEQ II study,
(b) Recommendations arising from analyses of SACMEQ II data, and (c) policy questions raised by SACMEQ country ministries of education. These questionnaires were refined by the SACMEQ scientific committee, then piloted in each SACMEQ country and refined further before they were administered.
- One important innovation in the development of questionnaires for the SACMEQ III study was introduction of a “Homework form” for pupils to take home. This consisted of questions to which the pupil might not know the answers (for example, parental education, estimates of travel distance to school, home possessions, whether or not their biological parents were alive) that parents, family members, or guardians could help in filling in. This considerably reduced the number of missing values in the SACMEQ III study compared with previous SACMEQ studies.
- Materials were translated into Kiswahili (Tanzania Mainland and Zanzibar) and Portuguese (Mozambique).
Ministry of Education
Data Entry was done using WinDEM (Windows Data Entry Manager) Software. Preliminary data cleaning involved checks on data to ensure it was clean before it was sent to the SACMEQ Coordinating Centre (SCC) for further checks an analysis and calculation of sampling weights. (See p12 of the NRC Manual - provided as external resources - for more detail on the process.)
Data Checking and Data Entry
The Mauritian NRC received the completed materials from the data collectors and kept these safely while they were being checked, entered into computers, and then “cleaned” to remove errors prior to data analysis. Data- checking involved the “hand editing” of data collection instruments by a team of trained staff. The staff checked that:
(i) All expected questionnaires, tests, and forms had been received,
(ii) The identification numbers on all instruments were complete and accurate, and
(iii) Certain logical linkages between questions made sense (for example, they had to verify if the two questions to School Heads concerning “Do you have a school library?” and “How many books do you have in your school library?” were answered consistently). Trained data capturers from Central Information Systems Division (CISD), supervised by the NRC, entered data into computers using the WINDEM software that was supplied by the SACMEQ Coordinating Centre. Data were “double entered” in order to monitor accuracy.
During December 2007, the SACMEQ Coordinating Centre organized a training programme for all NRCs. The teams were led step-by-step through the required data cleaning procedures that they were to follow in their respective countries. At individual country level, NRTs followed a “cyclical” process whereby data files were
cleaned by the NRT and then emailed to the Coordinating Centre for checking and then emailed back to the NRC for further cleaning. The entire data cleaning process in Mauritius lasted from January 2008 to August 2009.
To clean the data, using the WINDEM software, the NRCs followed specific directions to
(i) Identify major errors in the sequence of identification numbers,
(ii) Cross-check identification numbers across files (for example, to ensure that all pupils were linked with their own Reading and Mathematics teachers),
(iii) Ensure that all schools listed on the original sampling frame also had valid data collection instruments and vice-versa,
(iv) Check for “wild codes” that occurred when some variables had values that fell outside pre-specified reasonable limits, and
(v) Validate that variables used as linkage devices in later file merges were available and accurate.
The quality of the data provided by the school heads, teachers, and pupils was examined in the following ways.
First, at the time of data collection, the data collectors who visited the schools verified, for example,
(a) The actual existence and conditions of the school resources such as library, school head office, and staff room, and
(b) The official school records about the information provided by pupils such as their gender, age, days absent, and whether or not their parents were alive.
Second, similar questions were included in the school head, pupil, and teacher questionnaires, and these helped to verify the responses given by the respondents during data cleaning. For example, a question on the existence of a class library was included in both the teacher and pupil questionnaires. Any inconsistencies between the responses of the school heads, teachers, and pupils were followed up by the national research coordinators (NRCs) and corrected during data cleaning.
The processes of generating pupil scores, competency levels, measure of school location, socioeconomic status and tabulations are outlined in the SACMEQ-III Project Results Working Document Number 1 available as external resources.
Director- Southern Africa Consortium for Monitoring Educational Quality (SACMEQ)
International Institute for Educational Planning - UNESCO
TERMS AND CONDITIONS FOR USE OF THE SACMEQ DATA ARCHIVE
The Southern and Eastern Africa Consortium for Monitoring Educational Quality (SACMEQ) Co-ordinating Centre (SCC <http://www.sacmeq.org/_legal/accept_new?destination=training-workshops>) has produced a data archive containing all information collected for SACMEQ's first three educational policy research projects (SACMEQ I, SACMEQ II, and SACMEQ III). This archive is now available online on the SACMEQ website so as to give bona fide researchers and students online access to SACMEQ data and documents.
The SACMEQ data sets have been developed at great cost and with the application of stringent quality controls. It is being made available to eligible users because it has a great potential to contribute to educational policy development beyond what has already been achieved in this respect through the reports written by the National Research Co-ordinators (NRC <http://www.sacmeq.org/_legal/accept_new?destination=training-workshops>s) and Deputy National Research Coordinators (NRC <http://www.sacmeq.org/_legal/accept_new?destination=training-workshops>s). It is expected that many researchers and students will wish to use the Data Archive for research, publications, and/or training purposes.
The Terms and Conditions serve two purposes. Firstly, they provide interested applicants with guidelines on how to access this valuable information resource. Secondly, they are intended to safeguard against the danger of users being unaware of the complexities of the data collection process and consequently arriving at misinterpretations that could lead to incorrect conclusions.
2.0 How can the user gain such access?
In order to obtain SACMEQ Data Archive for any of the SACMEQ school systems, the applicant should follow these steps:
2.1 Read and Agree to these "Terms and Conditions for the Use of the SACMEQ Data Archive."
2.2 Complete an online application form.
3.0 What rules govern the use of the SACMEQ data archive?
3.1 The Data Archive is the outcome of expensive and time-consuming activities of the staff of the represented Ministries of Education spread over many years. For this reason, the SACMEQ Ministries of Education described in the Data Archive should:
3.1.1 be notified by the SACMEQ SCC <http://www.sacmeq.org/_legal/accept_new?destination=training-workshops> of any request for data;
3.1.2 have an opportunity to review reports based on the data archive so as to correct any gross errors before they are published; and
3.1.3 satisfy themselves that the data have been used in such a manner that they contribute positively to the development of relevant education policies in relevant SACMEQ member countries.
3.2 It is the National Research Coordinators (NRC <http://www.sacmeq.org/_legal/accept_new?destination=training-workshops>s) and Deputy National Research
Coordinators (DNRCs) who have spearheaded the collection and compilation of SACMEQ data. In acknowledgment of their efforts, the applicant(s) will be required to invite the relevant country's National Research Coordinator to participate in the study associated with the use of the data. Where an individual other than the NRC <http://www.sacmeq.org/_legal/accept_new?destination=training-workshops> or DNRC is co-opted, the relevant NRC <http://www.sacmeq.org/_legal/accept_new?destination=training-workshops> and DNRC shall be given the first right of refusal.
3.3 This provision does not apply in situations where the SACMEQ Data Archive is used purely for purposes of individual academic research by a student, and where the results are not intended for publication.
3.4 All relevant NRC <http://www.sacmeq.org/_legal/accept_new?destination=training-workshops>s and DNRCs will be informed by the SCC <http://www.sacmeq.org/_legal/accept_new?destination=training-workshops> about the recipients of the Data Archive.
3.5 SACMEQ provides the SACMEQ Data Archive to applicants on the basis of the intended use stated in the application. The applicant, therefore, should not use the data for any purpose other than the one stated in the application. Should the applicant(s) wish to use the data for a purpose other than that stated in the agreement, then he/she/they must first secure the written approval of SACMEQ before he/she/they proceed to do so.
3.6 SACMEQ data are provided for the sole and exclusive use of the applicant specified in the agreement. The successful applicant should, therefore, not share the SACMEQ Data Archive with, or pass it on to, any other unauthorized person(s).
3.7 The authorized user shall take responsibility for the safe custody of the SACMEQ Data Archive and also take reasonable steps to ensure that no unauthorized persons gain access to it.
3.8 The authorized user shall give due credit to SACMEQ for providing the Data Archive by providing written acknowledgement of this in any publication emanating from their use.
3.9 As the Data Archive remains the property of the SACMEQ, no other person(s), including the successful applicants or the member Ministry, shall re-distribute or offer for sale the SACMEQ Data Archive.
3.10 All reports based on the SACMEQ Data Archive have to secure the written approval of the SCC <http://www.sacmeq.org/_legal/accept_new?destination=training-workshops> prior to the publication in order to confirm compliance to our terms and conditions, and also to ensure that there is no misunderstanding or misinterpretation of the data.
3.11 Once authorization has been granted to access the archive, you will see a link on the website which will take you to the Data Archive.
3.12 All relevant NRC <http://www.sacmeq.org/_legal/accept_new?destination=training-workshops>s will be informed by the SCC <http://www.sacmeq.org/_legal/accept_new?destination=training-workshops> about the recipients of the SACMEQ Data Archive.
3.13 Full acknowledgement of the source of the data (including reference to the SACMEQ Data Archive) must be given whenever the data are used.
3.14 A copy of any published article or report based on the SACMEQ Data Archive must be provided free of charge to (a) the SACMEQ Co-ordinating Centre, and (b) the Ministry(ies) of Education from whose data the report has been generated.
Citation requirement is the way that the dataset should be referenced when cited in any publication. Every dataset should have a citation requirement. This will guarantee that the data producer gets proper credit, and that analytical results can be linked to the proper version of the dataset. The Access Policy should explicitly mention the obligation to comply with the citation requirement. The citation should include at least the primary investigator, the name and abbreviation of the dataset, the reference year, and the version number. Include also a website where the data or information on the data is made available by the official data depositor.
Southern and Eastern Africa Consortium for Monitoring Educational Quality. SACMEQ Project III 2007 [dataset]. Harare: SACMEQ [producer], 2007. Paris: International Institute for Educational Planning, UNESCO [distributor], 2007. Ref. MUS_2007_SACMEQ-III_v01_M. Dataset downloaded from [URL] on [date].
Disclaimer and copyrights
The user of the data acknowledges that the original collector of the data, the authorized distributor of the data, and the relevant funding agency bear no responsibility for use of the data or for interpretations or inferences based upon such uses.
Copyright, Southern and Eastern Africa Consortium for Monitoring Educational Policy
Director- Southern Africa Consortium for Monitoring Educational Quality
International Institute for Educational Planning - UNESCO
International Institute for Educational Planning
United Nations Educational, Scientific and Cultural Organization (UNESCO)