Case Study

Purdue University, in Indiana, is a public research university with over 44,000 students. It celebrated its 150th anniversary in October 2019.

One of the courses offered by Purdue is Design Thinking in Technology, in which first-year students develop design-thinking skills as they work through several iterations of practical design.

One particular task that learners often struggle to grasp is a Point-of-View statement (POV), where students must approach the design challenge from the end user’s perspective. A good POV gives students focus for their work, as it means they have successfully identified a user, that user’s needs, and relevant insights.

Dr. Scott R. Bartholomew, an engineering/technology educator at Purdue, aimed to explore whether giving the students an opportunity to view and evaluate work from previous students, would prove to be an effective way of helping students to understand the POV task and therefore improve their work.

In a traditional classroom setting, the work produced by students is then marked by teachers, after which students receive a grade. This can lead to a disconnect between learners and the grades they achieve.
Finding a tool that brings together students for an effective and beneficial collaborative learning experience, whilst also freeing up teachers so that they can better facilitate the learning and utilise their pedagogical expertise.
Discovering a cloud-based, flexible solution that would be simple to set-up and implement.

RM Compare is a new interactive digital assessment solution from RM. It is designed to enable an easy and flexible approach to formative assessment and collaborative learning.

The Adaptive Comparative Judgement (ACJ) technology is based on the Law of Comparative Judgement, which proves that people are better at making comparative, paired judgements, rather than absolute ones. This approach is particularly beneficial in subjects where work is more open-ended, such as English, Art, or Design Technology. As the software is cloud-based, it is easy to introduce to classrooms or groups of students at any size or scale.

In the largest-ever study of Adaptive Comparative Judgement, half of the 550 first-year students in the Design Thinking in Technology course at Purdue University were randomly selected to either participate in an ACJ experience using RM Compare, or to a control group that participated in the usual classroom activities.

The group selected to use ACJ were given the opportunity to view work submitted by a previous set of students who had undertaken the Point-of-View statement task, with a different assignment brief. This was achieved using RM Compare, which displays two pieces of anonymised work at a time. Students then evaluated the work and chose which they thought more closely met the criteria. Meanwhile, the control group took part in more traditional classroom activities where they directly shared work, thoughts, and feedback with peers and participated in teacher-led discussion. To ensure validity of the study, as students completed their design portfolios, all other elements of the course were unchanged, including being taught by the same lecturers, in the same classrooms.

The feedback from students showed they found the activity enjoyable, and it benefited them to see and analyse such a wide range of work.

The ACJ technology uses a unique algorithm to intelligently select and pair pieces of work based on previous judgements, displaying similarly ranked work side-by-side to achieve a finer level of detail in comparisons. The intelligent pairing reduces the time it takes to accurately place a piece of work on the professional consensus rank order it then generates.

The resulting rank order – which displays where each piece of work sits in terms of attainment within the cohort – can then be used as an informative learning tool, giving the teacher an opportunity to engage students in a discussion about the results and learnings of the exercise.

Seven students out of the top ten highest-performers on the task were part of the group that used RM Compare. Additionally the rank order clearly showed that the students who were given the opportunity to use RM Compare to evaluate earlier work performed significantly better than their peers who used traditional classroom methods.

The simple intervention had such a significant positive impact on the attainment of students in the ACJ group because it exposed them to a much broader set of work by former students, helping them to better identify what a good – or exemplar - piece of work consisted of. The action of deciding which piece of work out of a pair is better than the other helps them to internalise the learnings from the exercise and the course in general.

These learnings are then further solidified as students verbalise their thoughts to provide feedback to peers. The feedback they receive, in turn, becomes a formative learning opportunity, where they can apply that guidance to their work.

“We found that students who had used RM Compare from the outset performed significantly better than their peers, despite it being a small and easy to implement intervention. We see it as a great opportunity to help our students learn through evaluation – it turns the assessment process into a learning experience. The students collaborate with peers and with teachers to successfully improve their grades.

As well as students helping to teach each other, the use of ACJ allows the teacher to utilise their expertise in the most efficient way: facilitating the learning, helping struggling students, guiding and coaching through the formative assessment process.

Interestingly, the use of ACJ technology did not just benefit either the highest or lowest achieving students. It actually boosted the attainment of all the students who used RM Compare as a learning intervention.

I would recommend this approach to educators at all levels of education, especially in open-ended subjects such as technology and design thinking. In this instance RM Compare was used by first-year students collaborating together, yet it would be possible to achieve similar positive results from young children right through to adult learning. The software has built-in scalability, so a similar exercise could be undertaken with any number of learners – a class of 20 students would benefit, as would a group of more than 500.”

Dr. Scott R. Bartholomew, PhD, Assistant Professor, Engineering/Technology Teacher Education

Transforming Assessment

Learning through evaluation: study of 550 students at Purdue University proves that Adaptive Comparative Judgement raises attainment

Purdue University

Introduction

Goals

Key Challenges

The solution

RM Technology

RM Assessment

About us

Customers

RM plc

Policy