Cosgrove, Judith
(2005)
Issues in the Interpretation of PISA in Ireland:
A Study of the Content and Design of PISA with Comparative Analyses of the Junior
Certificate Examinations and TIMSS (1 Volume).
PhD thesis, National University of Ireland Maynooth.
Abstract
The results of the Programme for International Student Assessment (PISA) have been
the subject of attention in media reports and ministerial speeches in Ireland since 2001.
Yet, to date, there has been no examination of the appropriateness of the data it yields to
inform educational policy. Analyses described in this dissertation were carried out to
identify aspects of the design and interpretation of the PISA assessments of reading and
mathematics which may be problematic and/or at odds with conclusions drawn. Three
major questions are addressed. First, what does PISA tell us about achievements of
students in Ireland? While PISA is intended to be used for educational improvement, the
fact that it does not purport to assess school-based knowledge and skills could be
problematic. In this context, the extent to which the PISA achievement measures are
similar to, or differ from, the national curriculum (Junior Certificate syllabus and
examinations) is examined. The PISA reading measure was found, by and large, to be
compatible with Junior Certificate English. However, the diverging philosophies
underlying PISA and Junior Certificate mathematics result in notable disparities
between the two assessments which pose challenges in interpreting the results (although
these disparities may serve a potentially valuable ‘enlightenment’ function and act as a
trigger for curriculum review). Both PISA and the Junior Certificate Examination were
found to be of very limited utility in describing performance at the low end of the
achievement distribution. Analyses of non-response in the Irish datasets for PISA 2000
and 2003 reveal significant bias arising from student (but not school) non-response.
This finding renders claims made by the OECD and Irish media about the
characteristics of low achievers problematic and strengthens the argument that PISA is
not well suited to describing characteristics of low achievers in Ireland. The second
question considered is what PISA tells us about the equity of achievement outcomes in
Ireland. Analyses of the between-school variance statistic were used to address this
question since is widely cited as an indicator of educational equity. However, when
inferences are being made on the basis of between-school variance, no cognizance is
taken of how sample design and other characteristics of PISA might have impacted on
its magnitude. This issue is of particular relevance in Ireland where studies have
revealed large achievement differences associated with class allocation within schools.
Furthermore, the nature of the achievement measure used (in particular, its curriculum
sensitivity) would also appear relevant to the interpretation of the significance of
between-school variance. Comparative analyses of the TIMSS 1995 and PISA data
indicated that between-'school' variance in Ireland is much larger when the sample
design involves the selection of intact classes (in TIMSS) rather than on the basis of
students' age (in PISA). Furthermore, school-dependent and curriculum-sensitive
measures tend to be associated with higher between-school variance. The third question
relates to what PISA tells us about the determinants of the achievements of Irish
students. This question is of some significance since the impact of schools’ social intake
on student achievement, and the extent to which school practice variables explain
achievement, are given prominence in the results of PISA and in other surveys both
nationally and internationally. A comparison of multilevel models of Irish student
achievement on PISA 2000, PISA 2003 and TIMSS 1995 using achievement on the
international tests and the Junior Certificate Examinations suggests that the impact of
social intake is larger when surveys use a sample design based on intact classes, while
school-dependent and curriculum-sensitive measures may be more sensitive to school
practice variables. In the conclusions, some policy implications are described, the
limitations which these findings place on the interpretation of results considered, some
improvements to the design of PISA that may overcome some of the limitations
suggested, and areas for future research proposed.
Item Type: |
Thesis
(PhD)
|
Keywords: |
Interpretation of PISA in Ireland:
Study; Content and Design; PISA; Comparative Analyses; Junior
Certificate Examinations; TIMSS; |
Academic Unit: |
Faculty of Social Sciences > Education |
Item ID: |
12361 |
Depositing User: |
IR eTheses
|
Date Deposited: |
03 Feb 2020 17:23 |
URI: |
|
Use Licence: |
This item is available under a Creative Commons Attribution Non Commercial Share Alike Licence (CC BY-NC-SA). Details of this licence are available
here |
Repository Staff Only(login required)
![View Item View Item](/style/images/action_view.png) |
Item control page |
Downloads per month over past year
Origin of downloads