Progress in International Reading and Literacy Study 2016

PIRLS is designed to provide valid and reliable measurement of trends in student achievement in countries around the world, while keeping to a minimum the burden on schools, teachers, and students. The PIRLS program employs rigorous school and classroom sampling techniques so that achievement in the student population as a whole may be estimated accurately by assessing just a sample of students from a sample of schools. PIRLS assesses reading achievement at fourth grade. The PIRLS 2016 cycle also included PIRLS Literacy-a new, less difficult reading literacy assessment, and ePIRLS-an extension of PIRLS with a focus on online informational reading.

PIRLS employs a two-stage random sample design, with a sample of schools drawn as a first stage and one or more intact classes of students selected from each of the sampled schools as a second stage. Intact classes of students are sampled rather than individuals from across the grade level or of a certain age because PIRLS pays particular attention to students’ curricular and instructional experiences, and these typically are organized on a classroom basis. Sampling intact classes also has the operational advantage of less disruption to the school’s day-to-day business than individual student sampling.

SAMPLE SIZE

For most countries, the PIRLS precision requirements are met with a school sample of 150 schools and a student sample of 4,000 students for each target grade. Depending on the average class size in the country, one class from each sampled school may be sufficient to achieve the desired student sample size. For example, if the average class size in a country were 27 students, a single class from each of 150 schools would provide a sample of 4,050 students (assuming full participation by schools and students). Some countries choose to sample more than one class per school, either to increase the size of the student sample or to provide a better estimate of school level effects.

For countries choosing to participate in both PIRLS and PIRLS Literacy, the required student sample size is doubled-i.e., around 8,000 sampled students. Countries could choose to select more schools or more classes within sampled schools to achieve the required sample size. Because ePIRLS is designed to be administered to students also taking PIRLS, the PIRLS sample size requirement remains the same for countries choosing also to participate in ePIRLS.

PIRLS STRATIFIED TWO-STAGE CLUSTER SAMPLE DESIGN

The basic international sample design for PIRLS is a stratified two-stage cluster sample design, as follows:

• First Sampling Stage. For the first sampling stage, schools are sampled with probabilities proportional to their size (PPS) from the list of all schools in the population that contain eligible students. The schools in this list (or sampling frame) may be stratified (sorted) according to important demographic variables. Schools for the field test and data collection are sampled simultaneously using a systematic random sampling approach. Two replacement schools are also pre-assigned to each sampled school during the sample selection process, and these replacement schools are held in reserve in case the originally sampled school refuses to participate. Replacement schools are used solely to compensate for sample size losses in the event that the originally sampled school does not participate. School sampling is conducted for each country by Statistics Canada with assistance from IEA Hamburg, using the sampling frame provided by the country’s National Research Coordinator.

• Second Sampling Stage. The second sampling stage consists of the selection of one (or more) intact class from the target grade of each participating school. Class sampling in each country is conducted by the National Research Coordinator using the Within-School Sampling Software (WinW3S) developed by IEA Hamburg and Statistics Canada. Having secured a sampled school’s agreement to participate in the assessment, the National Research Coordinator requests information about the number of classes and teachers in the school and enters it in the WinW3S database.

Classes smaller than a specified minimum size are grouped into pseudo-classes prior to sampling. The software selects classes with equal probabilities within schools. All students in each sampled class participate in the assessment. Sampled classes that refuse to participate may not be replaced.

For countries participating in both PIRLS and PIRLS Literacy, students within a sampled class are randomly assigned either a PIRLS or PIRLS Literacy booklet through a booklet rotation system. This is done to ensure that PIRLS and PIRLS Literacy are administered to probabilistically equivalent samples. In countries taking part in ePIRLS, all students assessed in PIRLS are expected to participate in ePIRLS.

STRATIFICATION

Stratification consists of arranging the schools in the target population into groups, or strata, that share common characteristics such as geographic region or school type. Examples of stratification variables used in PIRLS include region of the country (e.g., states or provinces); school type or source of funding (e.g., public or private); language of instruction; level of urbanization (e.g., urban or rural area); socioeconomic indicators; and school performance on national examinations.

In PIRLS, stratification is used to:

• Improve the efficiency of the sample design, thereby making survey estimates more reliable

• Apply different sample designs, such as disproportionate sample allocations, to specific groups of schools (e.g., those in certain states or provinces)

• Ensure proportional representation of specific groups of schools in the sample School stratification can take two forms: explicit and implicit. In explicit stratification, a separate school list or sampling frame is constructed for each stratum and a sample of schools is drawn from that stratum. In PIRLS, the major reason for considering explicit stratification is disproportionate allocation of the school sample across strata. For example, in order to produce equally reliable estimates for each geographic region in a country, explicit stratification by region may be used to ensure the same number of schools in the sample for each region, regardless of the relative population size of the regions.

Implicit stratification consists of sorting the schools by one or more stratification variables within each explicit stratum, or within the entire sampling frame if explicit stratification is not used. The combined use of implicit strata and systematic sampling is a very simple and effective way of ensuring a proportional sample allocation of students across all implicit strata. Implicit stratification also can lead to improved reliability of achievement estimates when the implicit stratification variables are correlated with student achievement.

National Research Coordinators consult with Statistics Canada and IEA Hamburg to identify the stratification variables to be included in their sampling plans. The school sampling frame is sorted by the stratification variables prior to sampling schools so that adjacent schools are as similar as possible. Regardless of any other explicit or implicit variables that may be used, the school size is always included as an implicit stratification variable.

SCHOOL SAMPLING FRAME

One of the National Research Coordinator’s most important sampling tasks is the construction of a school sampling frame for the target population. The sampling frame is a list of all schools in the country that have students enrolled in the target grade and is the list from which the school sample is drawn. A well-constructed sampling frame provides complete coverage of the national target population without being contaminated by incorrect or duplicate entries or entries that refer to elements that are not part of the defined target population.

A suitable school measure of size (MOS) is a critical aspect of the national sampling plan, because the size of a school determines its probability of selection. The most appropriate school measure of size is an up-to-date count of the number of students in the target grade. If the number of students in the target grade is not available, total student enrollment in the school may be the best available substitute.