Case Study | NCFE

NCFE is an educational charity in vocational and technical learning. Just like RM, NCFE is a long established education organisation focused on enhancing learning and improving life chances.

Both organisations also share a deep passion and understanding of innovation and have worked together for a number of years in various capacities.

In this latest project, the RM Compare team worked with NCFE to try to approach a particular assessment challenge with a new holistic approach using Adaptive Comparative Judgement.

NCFE implemented RM Compare’s Comparative Judgement (CJ) system for both the AIF and Aspiration Award’s processes.

The platform’s key features included:

Paired comparisons of applications
Automated rank ordering based on multiple judgements
User-friendly interface for judges
Comprehensive data analysis and reporting tools

Project 1: Assessment Innovation Fund (AIF) Evaluation

The NCFE Assessment Innovation Fund (AIF) is an initiative to support and develop innovative approaches to assessment in education. Its aims are to break boundaries in assessment practices, support and pilot new ideas for the future of assessment, build trust and confidence in innovative assessment methods, and add value to stakeholders including learners, educators, employers, and government.

40 applications received, plus 4 seed scripts from previous applications
61 volunteer judges, each tasked with making 10 judgements

Project 2: Aspiration Awards Shortlisting

The Aspiration Awards are about honouring the success of learners, apprentices, educators, support staff and educational organisations across the UK. There are multiple awards including Against All Odds, Apprentice of the Year, Centre of the Year and Learner of the Year.

379 entries across six different categories
41 volunteer judges distributed across categories
10 judges for categories with many entries, 5 for those with fewer
Each judge set to make 50-60 judgements in their assigned category

Result 1: Dramatic Time Saving

The CJ method substantially reduced the overall evaluation time in both trials. For the AIF trial, the time spent on evaluation decreased by 58%, despite more judges and more application reviews. Similarly, in the Aspiration Awards, the total time spent on shortlisting was reduced by 43%, again with significantly more reviews of each application and more judges. The additional eyes on each application gave teams a high degree of confidence in the process compared to previous methods.

Overall Savings

Combined Savings
across both processes

Based on average employee cost

Result 2: Enhanced stakeholder engagement

Feedback from stakeholders was overwhelmingly positive. Judges found the CJ process user-friendly and efficient. They appreciated the simplicity and the ability to manage their judgements around other commitments. The increased engagement, with more judges involved than in previous methods, contributed to a more inclusive and comprehensive evaluation process. Administrators also found the setup and management of CJ sessions straightforward, with time savings and significantly improved visibility of progress.

Result 3: High reliability

The findings from both trials indicate significant improvements in time efficiency, reliability, and stakeholder satisfaction. Both trials demonstrated high reliability scores, indicating consistent and dependable judgements. In the AIF trial, the reliability score reached 0.73, which is considered acceptable given the breadth of applications. For the Aspiration Awards, the reliability scores were even higher, averaging above 0.80 across all categories. CJ provided direction to, and evidence for, the moderation processes for both activities, further enhancing reliability.

Transforming Assessment

How Adaptive Comparative Judgement further enhanced a commitment to assessment innovation and excellence

Case Study | NCFE

Introduction

The challenge

The projects

The results

Results summary

Conclusion

Next steps

“The findings from both trials indicate significant improvements in time efficiency, reliability, and stakeholder satisfaction.”

RM Technology

RM Assessment

About us

Customers

RM plc

Policy