Some practical reflections on its use in practice
I was first made aware of Comparative Judgement as a method of assessment last year, through one of David Didau’s informative blogposts. I had always meant to get around to using it, but was put off by a fear of using technology. I have regularly compared scripts when awarding marks, and have on occasion sought to put together some sort of order before being brought back to the use of nomoremarking.com by my Deputy Head, and fellow A-level history teacher, to mark some Y12 mock essays.
Having had some new, functional photocopiers installed with a scanning function, I was willing to press ahead. I shall outline the process below for the uninitiated and then offer a simple evaluation of its value below. I’ll probe these thoughts more deeply later in the week.
- Scan in the exam scripts. Really easy if you have a ‘scan to USB’ function on your photocopiers. I’ve become a dab hand at this. You’ll want to use an easy code (like P12 for the twelfth student in 8P) to name the files, rather than perhaps typing in all of their names. Each essay/piece of work needs to be scanned separately. It took me about 15 minutes to scan in 46 sixth form mock scripts.
- Upload the scripts to a new task on nomoremarking.com which is free to use.
- Get judging. It took a Luddite such as myself a little while to find this function. Bizarrely, the web address to access the scripts is located in a section called ‘judges’ but once there you simply click left and right, depending on which script is better in your opinion. Nomoremarking recommends going with your gut and taking less than 30seconds to make a judgement. In practice, this was true of some Y8 essays I’ve compared, but sixth form essays took an average of three minutes to judge.
- The data coming in is easy to read. You are provided with a downloadable readout of the rank order of your pupils. It also comes with an ‘Infit’ score to consider which essays the software is less confident in placing. This is often where you have invited multiple judges, and you have perhaps implicitly disagreed on its value.
- Apply some marks. I have been less sure of this. However, I’ve read a selection of essays, found some on the level boundaries, applied marks, and then distributed the marks evenly throughout the levels.
On essays where the Infit score is above 1.0 (indicating unreliable judgements) we’ve had some really interesting discussions about the merits of the essays, what we should be looking for and then manually awarded marks using an exam board mark scheme. I think it is clearly going to be valuable if you bank scripts from year to year with marks you are confident with, and feed them in – this should save time in awarding marks, if you have essays with firm marks already in the mix.
Dare I say it, judging essays has become fun. The clicking gamifies marking and I’m in a scramble to meet my marking quota. we have found that multiplying the number of scripts by 3 to determine the total number of judgements that need to be made, and evenly dividing this up between the team of markers works fine. In practice, essays are being compared against others 8 times there, and we’re achieving a reliability score of over 0.8 which David Didau says is the goal, and in excess of national examinations.
|Marks are awarded with great confidence, and a reliable set of data on the rank order of class is valuable for a range of accountability measures & considering further interventions.||It is difficult to overcome the urge to write individualised comments on essays. Students (and SLT?) need to expect feedback where this isn’t the case. This feeds in with Christodoulou’s recent work on separating out formative & summative assessments.|
|It’s quick. Doesn’t sound like it, but marking those mocks could easily have consumed 8 days at 15-20 minutes an essay. At 3 minutes an essay, plus scanning, plus determining marks (30 mins when you have no known marks within the judgement process) is significantly quicker.||Transforming electronically judged essays into generic feedback for pupils requires careful thought. I’m still refining this.|
|There is less of a marking bias. Especially if you ask pupils to submit essays with a code (see part one above) rather than naming them.||Essays that ‘miss the wash’ are troublesome to reliably fit in the process. This is probably more frustrating rather than the end of the world.|
|I have thought much more carefully about what I’m really looking for in essays. I think this has led me to be clearer, already, with my classes about how they need to develop their essays.||Getting an entire team on board with this might be more difficult than using the software individually. If you’re marking procedure is out of step with other staff, as a head of department, you can still have little confidence in the reliability of marks generated.|
How I intend to develop my use of comparative judgement further
- Ask students to highlight key areas of the script. This might involve showing the mark scheme, and asking them to pick out the five sentences they most want the examiner to see. This should speed up comparisons. Before I stuck my first essays through the process, I had already put formative comments on them. These were a useful aid in passing comment.
- Banking essays for next year with secure marks attached to them. This should eliminate significant amounts of time transforming the rank order into marks.
- Get students to submit work electronically. I am in the midst of getting KS3 to do this with an outcome task to a unit of work. I’m not sure how valuable this will be. Paper, pen and scanning seems to be less hassle, so far.**
- Learn what this anchoring business is which seems to be taking comparative judgement to the next level by connecting subsequent pieces of work together. If I get to the bottom of this, I’ll blog on it.
Comparative judgement seems to be a valuable tool for making summative judgements on the quality of pupils’ work. It does not replace marking, or feedback but these should be steps on the road towards a final piece of work. This is where the comparative judgement bit fits in.
Your thoughts and questions are invited.
** Update – I have now discovered that nomoremarking.com will not accept word documents. They need to be PDF files which throws this plan of mine out of the window.