Transforming Assessment
How the ISEB used adaptive comparative judgement to assess a new interdisciplinary qualification
Case Study | ISEB
Background
The Independent Schools Examinations Board (ISEB) is one of the world’s most recognised, respected and well stablished assessment organisations. Celebrating their 120th anniversary in 2023, they have helped millions of students at various stages of their learning journey in over 50 countries and continue to innovate to meet the needs of a changing assessment world.
In 2022, the ISEB launched their ground-breaking ISEB Project Qualification (iPQ) that adopts an interdisciplinary approach to focus on student-led, project-based learning. However, traditional assessment techniques proved challenging when standardising the diverse range of items produced.
So, the ISEB set RM a challenge to explore a way to reliably use teacher assessment within their new interdisciplinary qualification. To provide focus, it was agreed that any solution needed to address 3 key challenges:
- It must have teacher assessment at its core
- It must reduce teacher workload
- It must support the new iPQ by encouraging the development of independent learning among students especially in the production of diverse responses
This world class collaboration produced an innovative solution to assessing the new iPQ qualification through the use of Adaptive Comparative Judgement (ACJ). In doing so, the iPQ was assessed with RM Compare in Summer 2024 and the two organisations continue to work together to realise the full potential of RM Compare and ‘when-ready’ assessments.
“We have been delivering innovative assessments since 1903. A lot has certainly changed in that time and we are proud of our ability to deliver for our customers in changing times. Working with RM will ensure that we can meet the next set of challenges with confidence and continue to deliver world leading qualifications.”
Julia Martin, CEO, ISEB
The assessment challenge
An emerging theme worldwide is the desire to increase the capacity of education systems to encourage independent learning among all students. To do this, many institutions want to take an interdisciplinary approach where a planned experience brings disciplines together in one coherent project so that learners can apply knowledge in new and different ways and tackle problems across a variety of subject areas.
In 2022, the ISEB launched its ground-breaking iPQ to help schools on the independent learning quest. This takes the form of a 15-20 hour project that asks pupils to identify a question, conduct research, develop their views, present their findings, and reflect on the learning process. The emphasis is on the process rather than the output, providing the pupil with real insights into their own ways of thinking and working.
Encouragingly, participating students took the opportunity to submit very diverse responses. However, this raised a significant challenge of item standardisation. The original model had been for submissions to be teacher assessed, supported by a detailed rubric. The ISEB would then moderate a sample of the work. From the outset, there were a number of concerns about this approach including teacher bias, subjectivity, inaccuracy, lack of standardisation, ‘teaching to the test’ and the overall process being time consuming.
The ISEB remained committed to the principle of teachers taking an active role in the iPQ assessment process. It was felt however that traditional remedies - for example enhanced rubrics, more staff training, or further layers of moderation - would inhibit, rather than encourage, good practice and could undermine the overall intent of the qualification.
Exploring together
Comparative Judgement (CJ) enables assessors to compare two pieces of work side-by-side and decide which is better against a holistic statement. This is particularly well suited to highly diverse and subjective responses. The comparative process is done multiple times by multiple assessors which can ensure a high level of reliability. RM Compare is a digital tool that makes the comparative judgement process more efficient through the use of an adaptive algorithm. This optimises the presentation of work to be assessed as a judging session progresses.
RM and the ISEB collaborated on a research project to see if a comparative approach could indeed be a better way of assessing the iPQ. 78 iPQ items were assessed by a pool of 9 judges. The judges were a mix of ISEB staff and practising iPQ teachers.
- Each item was seen between 9 and 11 times by the judging pool
- Each judge completed 46 judgements. There were a total of 9 judges
- The average time to make a judgement was 88 seconds (total time to complete the task ranged from 22 minutes to 111 minutes)
- The estimated time to mark an iPQ with the rubric was 15 minutes
“The iPQ allows us to award the process, not the outcome. Skills-based holistic judgements encourage diverse, creative and innovative projects that can be personal to a child and show the unique character of each school. We have seen children start small businesses, design products, fight for social justice and create their own podcasts to name just a few.”
Julia Martin, CEO, ISEB
Learning together
Reduce teacher workload
Using a comparative approach dramatically reduced the time needed to assess and standardise the iPQ. As demonstrated in Table 1, it produced a workload saving of over 50% and with no further need for moderation.
| Total scripts | Decision per script | Total decisions | Decisions per hour | Total hours | Hrs per worker | |
|---|---|---|---|---|---|---|
| Marking | 78 | 1 | 78 | 4 | 19.5 | 2.20 |
| CJ | 78 | 10 | 390 | 41 | 9.51 | 1.05 |
Table 1: Workload calculator
Empower assessors
The judging process produced a stable rank with very high levels of reliability (demonstrated in Figure 1) indicating that teacher assessment could be retained as a key element of the iPQ. There was a high level of coherence between the diverse judging pool.
The CJ approach, where the work is seen multiple times by multiple judges, provided the confidence to participants that it was a better reflection of the true value of the work and could mitigate against assessor bias, subjectivity and inaccuracy.
As a point of interest we were able to see where previously marked work (Distinction, Merit, Pass) was ranked in the comparative process. The limited correlation was not unexpected but did encourage further reflection especially around matters of curriculum and pedagogy.
Provide a unique solution to a new market
The feedback from the project participants was overwhelmingly positive and was seen as a valid approach to assessment that would better support the intent of the new iPQ qualification than the current approach. Judges enjoyed the comparative experience and felt that it helped them to better understand what good quality work looked like. The teachers who took part felt that this new tacit knowledge would better help them to support learners.
What next?
The project has given the ISEB the confidence to use RM Compare as the future assessment solution for the iPQ. Summer 2024 has seen all iPQ projects assessed using ACJ. The intention is to encourage teachers to be a significant part of the judging pool.
At the same time the iPQ will be providing participating schools with the new RM Compare Manual On-Demand system. This allows teachers and learners to compare work to an established standard at any point to monitor progress and better understand what good looks like.
A key driver of this new approach is to facilitate positive curriculum and pedagogical change that will help teachers to produce even better outcomes for their students.
“At RM, we take pride in working with our customers to innovate. This initial research provides an exciting demonstration of how to tackle some of our education systems challenges. We look forward to driving this work forward with the ISEB.”
John Baskerville, Managing Director, RM Assessment
Beyond 2024, RM and the ISEB are committed to realising the full potential of RM Compare On Demand, including the possibility of ‘when-ready’ assessment.
Find out more about how the ISEB is driving innovation in assessment at https://www.iseb.co.uk
“The outcomes from our initial research are tremendously exciting and offer a real opportunity to solve some of the assessment’s biggest challenges. Working with RM we have the opportunity to help our teachers and learners to meet the demands of a fast changing world with confidence.”
Julia Martin, CEO, ISEB
RM is working with innovative assessment organisations around the world.
Interested in finding out more?
