Cross-national comparisons of responses to survey items are often affected by response style, particularly extreme response style (ERS). ERS varies across cultures, and has the potential to bias inferences in cross-national comparisons. For example, in both PISA and TIMSS assessments, it has been documented that when examined within countries, higher mean student academic achievement is associated with more positive mean student attitudes towards science; between countries however, higher mean student academic achievement is associated with more negative mean student attitudes (Buckley, 2009; Bybee & McCrae, 2007; Loveless, 2006). While such a result is theoretically possible, it has been widely speculated as being due to response style heterogeneity across countries.

In order to evaluate the possibilities that response style differences may affect the association between academic achievement and attitudinal measures in cross-national assessment, the current study proposes a multilevel multidimensional IRT (MMIRT) model for the measurement and control of ERS. To implement the MMIRT with the PISA 2006 science achievement and attitudes assessment, a two-stage estimation algorithm is applied. Real data analyses using the MMIRT appear to affect the unexpected country level correlations between science achievement and attitudes. Results are compared with those obtained by Buckley (2009) who analyzed the same dataset but applied an ad hoc method as well as a scale heterogeneity model proposed by Rossi, Gilula, and Allenby (2001), for measuring response style and controlling bias. The comparison demonstrates some of the unique characteristics of the MMIRT approach in modeling response style in cross-national assessments. Simulation analyses in the dissertation further substantiate the estimation accuracy of the MMIRT approach involving data structures resembling real data. Simulation analyses are also used to investigate whether the psychometric characteristics of scales play a role in response style trait estimation.

As an important goal of PISA is to better understand cross-cultural differences both in science achievement and attitudes towards science, the current application not only provides results that speak to the purpose of PISA but also helps clarify differences between methodologies that can be used to account for response style.