Going Digital:
How one province in Canada began the transition from paper to digital in high-stakes assessment
Case Study
New Brunswick Department of Education and Early Childhood Development
Introduction
In April 2020, the Assessment, Analysis and Design Services Branch of the New Brunswick Department of Education and Early Childhood Development approached RM, as they wanted to transition their English Language Proficiency Assessment (ELPA) to a digital platform. New Brunswick is one of Canada’s 10 provinces. Located on Canada’s East coast, and with a population of just under 800,000, it is unique amongst the provinces in having both English and French as its official languages.
The ELPA assesses students’ reading skills against end of grade 8 provincial curricular standards. It consists of 50 multiple choice items based on literary and informational texts and is a graduation requirement. It is administered in schools throughout New Brunswick and in some international schools as far afield as China. Previously, the ELPA was presented in booklet format which required a logistical solution to distribute and collect. Data obtained from the ELPA provides information about individual student achievement as well as school, district, and provincial trends.
After a thorough procurement process, New Brunswick partnered with RM to transform the delivery of their ELPA, with an ambition to extend the transformation into mathematics, French reading, and scientific literacy in a subsequent phase. RM implemented its SaaS-based modular digital assessment platform, enabling New Brunswick to make an immediate start on the transformation programme by organising the registration of students, authoring test items in English and French, and delivering the ELPA in all high schools. New Brunswick was immediately able to realise considerable operational efficiencies with the removal of the logistical problem from booklet distribution.
Key challenges
- Move exams from a paper-based delivery system to a computer-based delivery format without compromising integrity
- Handling significant political and public scrutiny
- Managing the complexities of implementing an e-assessment system in two official languages (English and French)
- A need to deliver the project in a short timescale
The solution
RM began implementation in January 2021 and progressed at pace with the first live assessments taking place in April 2021. The success of the ELPA examinations in 2021 led New Brunswick to extend the use of RM’s digital assessment platform, to their April 2022 exam sessions including tests in French reading, mathematics, and scientific literacy.
[The adoption of RM’s digital assessment platform] “has been one of the most positive things we have rolled out to schools that I’ve been a part of.”
Inga Boehler, Assessment Services Director
The implementation process
Pace and experience
New Brunswick was seeking an experienced partner who could help them successfully move from paper to digital at pace. They selected RM, a UK based company with a well-established record of accomplishment in the delivery of assessment solutions. Using their end-to-end platform, Assessment Master, RM were able to configure a tailored experience without the risk of delays in development. The quick launching of an instance of their platform in Canada for New Brunswick meant that all relevant data security regulations could be met. The project started with a collaborative discovery phase where RM and New Brunswick worked together to ensure a mutual understanding of requirements and RM’s platform functionality.
Once the discovery phase was complete, New Brunswick proceeded at pace through piloting, non-live testing and then through to project go-live with RM providing a dedicated project team in support during each phase.
A key element to success was the close collaboration between New Brunswick and RM in extensive scenario planning to build confidence in the run-up to live service.
Close collaboration
The first step to building a trusted partnership in a project of this size and working at this pace is to appoint an experienced resource team. This project illustrated the benefits of strong working relationships in the core team - the Project Manager and Solution Architect from RM, with the Project Lead at New Brunswick.
With the COVID-19 pandemic at its height at the time of the project, there were restrictions, so a schedule was prepared to accommodate online meetings. A mixture of semi-formal updates between the core team and more formal meetings with the wider New Brunswick operational team became the communication model for the project. Although face-to-face is always preferred, the online model proved effective, particularly in planning and monitoring to ensure that project go-live happened on time and on budget.
The onboarding process
Early engagement, the right functionality, political sensitivity and communication strategy
The feedback loop is an essential element of successful implementation, particularly with a new partnership. RM always plans opportunities to gain user feedback throughout project delivery. In this case RM recommended early engagement with schools and a cohort of students during a non-live pilot. This provided an opportunity for critical feedback into how item templates would best work for students, and what functionality was essential for teachers.
RM also provided New Brunswick with a testing environment to author items and offer practice tests. The auditable trail of actions created within the environment gave RM’s Solution Architects and the wider implementation team excellent insight into how users interacted with the platform – and allowed RM to anticipate queries that might be forthcoming.
From previous implementations for General Qualifications customers, RM also understood the importance of practice tests to enable students to familiarise themselves and get comfortable with the digital exam format – so they ensured this would be available for students prior to the roll-out of the assessments.
The combined early engagement work alerted RM to political issues surrounding exam practices and allowed both parties the opportunity to plan for all potential scenarios. Teaching staff in New Brunswick are highly organised and dedicated to their students’ best interests, so understandably there was some sensitivity around changes to assessment processes. RM made it a priority to work proactively, assisting the customer’s project team to manage these concerns appropriately.
Working in a bilingual province
Bilingualism was a principal factor in the authoring and delivery of assessments in New Brunswick. In the second session in April 2022, the types of assessment were extended to include French comprehension, mathematics, and scientific literacy. It was imperative that New Brunswick had the option to translate items into French. For this requirement, RM enabled functionality within the system so that students could choose what language questions were presented in and could even switch between languages as the assessment progressed.
Key benefits for Amplify:
Public sector organisations typically have a legal requirement to ensure that users with special educational needs are catered for. The assessment platform has accessibility options by design, and from the inception of the project, New Brunswick’s item authors were able to utilise options such as text to speech and specialised fonts. The impact of accessibility platform enhancements is powerfully summarised by the comments of a special educational need’s student: ‘I would normally have to take these assessments in a separate room, but I can now sit my exam with everyone else’.
“I now realize what a burden those booklets were to move in and out of the vault, constantly counting to make sure they were all there; what a break it will be for those who used to organize the booklets coming back to your department!”
School Administrator
The outcome
RM has been working to develop the adoption of innovative technologies in the education space for nearly 50 years. This experience proved invaluable when engaging with New Brunswick’s schools in designing a communications strategy that would be most effective, and when advising New Brunswick on the level of functional accessibility that works best in an educational environment.
Additionally, RM used its knowledge of educational processes in undertaking scenario planning exercises to gauge any concerns of New Brunswick teaching staff. In doing so, RM took the teachers, who were crucial to the success of the project, along on the journey. RM created essential training, communications and business continuity procedures that empowered staff to confidently deal with interruptions to service that power outages or connectivity failures might cause. RM’s knowledge and experience, combined with close collaboration with New Brunswick’s team, led to the successful delivery of 45,000 onscreen assessments to students across grades 4-10, including the English Language Proficiency Assessment (ELPA) over a 5-month period.
“They all found the system easy to navigate, and the texts were fair and varied.”
School Administrator
RM is working with innovative assessment organisations around the world.
Interested in finding out more?
Contact us for a conversation