Report on Predictability in the Leaving Certificate

Mon, May 11th, 2015 9:36:30 am

STATE EXAMINATIONS COMMISSION

PRESS RELEASE

Publication of Report on Predictability in the Leaving Certificate

2 May, 2015

Following on from last Wednesday’s launch by the Minister for Education and Skills of the report Supporting a better Transitions from Second Level to Higher Education, the State Examinations Commission has today published an independent research report on a study of predictability in the Leaving Certificate.

A commitment to address any problematic predictability in the Leaving Certificate examination was one of three key directions identified by a group (the Transition Reform group) established by the DES to facilitate collaboration between the key education partners at second level and higher education in progressing initiatives aimed at improving transition from second level to higher education.

In response to this commitment, the SEC commissioned an independent external evaluation of predictability in the Leaving Certificate examinations by the Oxford University Centre for Educational Assessment (OUCEA), under the direction of Professor Jo-Anne Baird, Pearson Professor of Educational Assessment and Director of the OUCEA, and in collaboration with Queens University, Belfast.

The research team have defined problematical predictability as follows:

A problematically predictable examination is one in which teachers and students can anticipate the test-taking conditions, performances required, question formats, and topics and scoring to the extent that undesirable effects upon the educational processes are pervasive. These include narrowing of the taught curriculum, superficial rote learning, drilling on test content and failure to develop a broad and deep understanding of a subject.

Their evaluation was conducted in three phases:

• a literature review and analysis of media commentary on the Leaving Certificate examinations;

• extensive analysis and research on examination materials in six subjects over 10 years;

• Teachers and Learners research including surveys of more than 1,000 Leaving Certificate candidates and fieldwork conducted in 17 representative schools across Ireland.

As there is little empirical evidence worldwide on the issue of predictability, the report notes that the research undertaken on the Irish system, being the first study of this breadth and depth on this issue, will be of interest to assessment organisations and researchers internationally.

The report of this evaluation is published today. The conclusions of and the main issues arising from the research are set out in the appendix.

Some key findings:

• Predictability and teaching to the test are fundamental concerns about high stakes examinations in many countries internationally

• Concerns about the predictability of the Leaving Certificate examination question content were not sustained by the findings of the research overall. None of the examinations was found to be very predictable in these terms

• Students who study broadly do better in the Leaving Certificate examinations than students who try and predict topics and narrow their learning by leaving out certain areas of the course

Moving beyond the issue of the predictability of examination question content to the broader issue of the kinds of learning that students engage in, the researchers did identify certain aspects of the examinations that could be beneficially addressed. These aspects include increasing the emphasis on the assessment of higher order thinking skills in the examinations. The researchers noted that this is a common concern internationally in relation to examinations, and that moving further in this direction as subjects are reviewed would be in keeping with international trends in assessment.

The Department of Education and Skills has asked the advice of the SEC on how to address the issues identified in the report. While some of the issues can be addressed in the short term, a longer term plan will have to be put in place to assess and address other. The SEC is considering the report and will forward its advice to the Department in the coming months.

Any changes to be introduced to the Leaving Certificate examinations in the future will be announced in advance in accordance with normal procedures and will be managed carefully to ensure fairness to candidates.

The report, along with four associated technical working papers, is available on the SEC website www.examinations.ie under the "Publications" heading.

Ends

Media enquiries to:

Cathal McDonagh Mary Downes

090-644 2770 or 087 2830355 090-644 2851

Email enquiries to sec.pressoffice@examinations.ie

Appendix

Conclusions of OUCEA Predictability Research and Issues for Consideration

Conclusions

i. Concerns about the predictability of the Leaving Certificate examination question content were not sustained by the findings of this research overall. None of the examinations was found to be very problematically predictable in these terms.

ii. It follows that media concerns about the predictability of the examinations were not supported by this research. As the media influences stakeholders’ views of the examinations, it is important for the currency of the Leaving Certificate that it is a trusted assessment. We therefore recommend that action be taken specifically to address the media portrayal of the examinations to maintain trust in these important national examinations.

iii. The predictability of examinations is a more subtle issue than question content alone, however. Concerns often relate to the effect of the examinations upon the kinds of learning that students engage in. Areas that could beneficially be addressed were noted in all six subjects investigated.

iv. In economics, French and DCG, subject specialist reviewers considered that the examinations were quite (not very) predictable overall.

v. Consideration should be given to placing more emphasis upon the assessment of higher order thinking skills in the examinations, in keeping with international trends in assessment.

vi. A more regular programme of revision of syllabuses is needed for the Leaving Certificate examinations to remain current. This is important for keeping up with improvements in assessment design (such as assessing more higher order thinking skills), as well as syllabus content.

vii. Marking schemes were thought to be lacking in transparency by the subject specialist reviewers, although not by the teachers and students. Any changes to the marking schemes to make them more transparent could have implications for the manageability of fixed grade boundaries (cutscores) in the Irish Leaving Certificate. Thus, there are decisions to be taken about whether marking schemes can be changed in this way whilst maintaining the current standard-setting system. The feasibility of this could be investigated in pilot studies.

viii. This research has added to the international research literature by providing a broader programme of research on the predictability of a national examination than was previously available. New scales for measuring student perceptions of the predictability of examinations were devised and the relationships between them and scores on the examination were investigated. To tackle wider issues of test preparation, the research took into account teachers’ and students’ views of the Leaving Certificate examinations and of preparing for them.

Issues for consideration in an Irish context

A number of issues that warrant consideration in an Irish context arise from this research. Some issues may be amenable to being addressed in the short term, but others may need a more fulsome consideration due to the culturally embedded nature of assessment systems. The possibility of unintended consequences needs to be considered and it may be that some changes need a longer term plan.

1. Consider how best to address the media narrative about predictability in the Leaving Certificate examinations, which has little basis in fact. The Leaving Certificate examinations are important life events for young people and have a great deal of pressure associated with them. Undermining the value of the Leaving Certificate with claims that the content of examination questions is overly predictable is unwarranted.

2. Contemplate monitoring the frequency with which questions focus upon particular topics and compare this with syllabus intentions.

3. Discourage drilling of students with pre-prepared examination answers that they do not fully understand. Although a minor issue in the findings of this research programme, this is a matter of concern where it arises. Understanding of examination formats is necessary, but teachers and students should be dissuaded from taking this too far as it will not prepare students for the future. High-stakes assessments encourage highly strategic (even if only in the short term) behaviours by students and teachers who want to get the best results. The DES, NCCA, SEC or even teachers are not fully in control of this issue. The best that can be done is to send the right signals.

4. Consider revising syllabuses more frequently and move towards the assessment of more higher order skills in the next round of revisions. Ensure that changes are reflected in the marking schemes as well as the syllabuses and question papers. Consider the ways in which students could narrow the intended curriculum and design the assessments so that this is not rewarded.

5. Maintain the transparency of the Leaving Certificate examination process, as students need to know what will be expected of them in the examination. To withdraw this would undermine the validity of the examinations. In no way do we mean this to imply that students should only be assessed on things they have already practised. Authentic learning and assessment involves being presented with novel questions and applications to some extent.

6. Consideration should be given to examining the strengths and weaknesses of the current and possible alternative approaches to standard setting. Unavoidable fluctuations in difficulty that arise in all public examinations from year to year need to be addressed through such a standard-setting system and the question is whether handling this in the current manner is the most effective approach for the Leaving Certificate.