Using Comparative Judgement

Some practical reflections on its use in practice

I was first made aware of Comparative Judgement as a method of assessment last year, through one of David Didau’s informative blogposts. I had always meant to get around to using it, but was put off by a fear of using technology. I have regularly compared scripts when awarding marks, and have on occasion sought to put together some sort of order before being brought back to the use of nomoremarking.com by my Deputy Head, and fellow A-level history teacher, to mark some Y12 mock essays.

Having had some new, functional photocopiers installed with a scanning function, I was willing to press ahead. I shall outline the process below for the uninitiated and then offer a simple evaluation of its value below. I’ll probe these thoughts more deeply later in the week.

The Process

  1. Scan in the exam scripts. Really easy if you have a ‘scan to USB’ function on your photocopiers. I’ve become a dab hand at this. You’ll want to use an easy code (like P12 for the twelfth student in 8P) to name the files, rather than perhaps typing in all of their names. Each essay/piece of work needs to be scanned separately. It took me about 15 minutes to scan in 46 sixth form mock scripts.
  2. Upload the scripts to a new task on nomoremarking.com which is free to use.
  3. Get judging. It took a Luddite such as myself a little while to find this function. Bizarrely, the web address to access the scripts is located in a section called ‘judges’ but once there you simply click left and right, depending on which script is better in your opinion. Nomoremarking recommends going with your gut and taking less than 30seconds to make a judgement. In practice, this was true of some Y8 essays I’ve compared, but sixth form essays took an average of three minutes to judge.
  4. The data coming in is easy to read. You are provided with a downloadable readout of the rank order of your pupils. It also comes with an ‘Infit’ score to consider which essays the software is less confident in placing. This is often where you have invited multiple judges, and you have perhaps implicitly disagreed on its value.
  5. Apply some marks. I have been less sure of this. However, I’ve read a selection of essays, found some on the level boundaries, applied marks, and then distributed the marks evenly throughout the levels.
    On essays where the Infit score is above 1.0 (indicating unreliable judgements) we’ve had some really interesting discussions about the merits of the essays, what we should be looking for and then manually awarded marks using an exam board mark scheme. I think it is clearly going to be valuable if you bank scripts from year to year with marks you are confident with, and feed them in – this should save time in awarding marks, if you have essays with firm marks already in the mix.

Dare I say it, judging essays has become fun. The clicking gamifies marking and I’m in a scramble to meet my marking quota. we have found that multiplying the number of scripts by 3 to determine the total number of judgements that need to be made, and evenly dividing this up between the team of markers works fine. In practice, essays are being compared against others 8 times there, and we’re achieving a reliability score of over 0.8 which David Didau says is the goal, and in excess of national examinations.

Strengths Weaknesses
Marks are awarded with great confidence, and a reliable set of data on the rank order of class is valuable for a range of accountability measures & considering further interventions. It is difficult to overcome the urge to write individualised comments on essays. Students (and SLT?) need to expect feedback where this isn’t the case. This feeds in with Christodoulou’s recent work on separating out formative & summative assessments.
It’s quick. Doesn’t sound like it, but marking those mocks could easily have consumed 8 days at 15-20 minutes an essay. At 3 minutes an essay, plus scanning, plus determining marks (30 mins when you have no known marks within the judgement process) is significantly quicker. Transforming electronically judged essays into generic feedback for pupils requires careful thought. I’m still refining this.
There is less of a marking bias. Especially if you ask pupils to submit essays with a code (see part one above) rather than naming them. Essays that ‘miss the wash’ are troublesome to reliably fit in the process. This is probably more frustrating rather than the end of the world.
I have thought much more carefully about what I’m really looking for in essays. I think this has led me to be clearer, already, with my classes about how they need to develop their essays. Getting an entire team on board with this might be more difficult than using the software individually. If you’re marking procedure is out of step with other staff, as a head of department, you can still have little confidence in the reliability of marks generated.

 

How I intend to develop my use of comparative judgement further

  • Ask students to highlight key areas of the script. This might involve showing the mark scheme, and asking them to pick out the five sentences they most want the examiner to see. This should speed up comparisons. Before I stuck my first essays through the process, I had already put formative comments on them. These were a useful aid in passing comment.
  • Banking essays for next year with secure marks attached to them. This should eliminate significant amounts of time transforming the rank order into marks.
  • Get students to submit work electronically. I am in the midst of getting KS3 to do this with an outcome task to a unit of work. I’m not sure how valuable this will be. Paper, pen and scanning seems to be less hassle, so far.**
  • Learn what this anchoring business is which seems to be taking comparative judgement to the next level by connecting subsequent pieces of work together. If I get to the bottom of this, I’ll blog on it.

 

Comparative judgement seems to be a valuable tool for making summative judgements on the quality of pupils’ work. It does not replace marking, or feedback but these should be steps on the road towards a final piece of work. This is where the comparative judgement bit fits in.

Your thoughts and questions are invited.

 

** Update – I have now discovered that nomoremarking.com will not accept word documents. They need to be PDF files which throws this plan of mine out of the window.

Advertisements

13 thoughts on “Using Comparative Judgement

  1. “Transforming electronically judged essays into generic feedback” – does there need to be generic rather than specific feedback? I would have thought the misconceptions that come up generally should be for the teacher to address through the lessons they teach.

    Like

    1. Yes, this is certainly a fair point, and one that I feel I am doing better as a result of this. However habit, and a fear of parent/pupil complaints means that I still need to be trained out of wanting to write an individualised comment on everything.

      How often and when is it most appropriate to provide individualised feedback? Presumably, from your point, you would be satisfied with putting essays into bands using CJ, highlighting strengths/weaknesses of essays that fall in each band to the students and then planning lessons that build on the core, common strengths & weaknesses that emerge from reviewing the work submitted?

      Liked by 1 person

      1. From a primary perspective – I wouldn’t have marked essays. I think that individual feedback was a bit pointless at the end unless we actually had time to edit the work (though SLT demanded it so I understand the feeling that it has to be done). Usually it was provided for smaller parts of the whole. I think what I was doing towards the end was a waste of time to be honest.

        However what you outline is exactly what I did when I taught undergraduates! It saved a lot of time marking but meant follow up lessons and revision sessions were more focused. But then again I didn’t have to deal with parents.

        Liked by 1 person

  2. Really interesting to read your reflections – the time saving is particularly interesting. Do feel free to use the QR Sheets which are now free for use – could help with reducing admin. Alternatively a number of schools ask pupils to save their Word documents as pdfs which is usually built in functionality to Word these days.

    Like

    1. Hi Chris, thanks for reading the blog and commenting. I was not aware that the QR sheets are now free – I will certainly experiment with this in the coming days/weeks.
      I have only just thought of getting students to directly save as PDF and I will give this a go, however I also found some students do not have access to the internet at home, or proper computer access so. This made it more difficult for some to submit work, and led to delays to my marking. I therefore think it is fairer, and on balance, administratively easier for me to do the scanning.

      I was discussing this with colleagues yesterday. There was a fear among some that it might one day become paid for software, or filled with pop-ups. I wasn’t quite so afraid, but could you allay the concerns of these teachers?

      Like

    2. We’re also going to be running a little ‘project’ where we are going to map out three possible ways that we could inject comparative judgement into departmental mark schemes and investigate which works best for us in practice. Would this be of interest to you?

      Like

  3. Two things we need Daisy to get happening with comparative judgement and no more marking.

    1. We are paperless. We do all work electronically. We don’t do scanning. Hard to convince staff of the time saving when they are shuffling paper at the photocopier. We need to get NMM working with LMS systems like Moodle.

    2. Feedback: While I understand the NMM is primarily designed to facilitate efficient accurate summative assessments we can’t resist the urge to give some formative feedback. Our English students in Level 12 do a text response in Term 1 and another in Term 3. We would use NMM to generate rank order of the essays in Term 1 but want to be able to add some formative feedback before students do their term 3 essay. While we could do this via other means it would be better to be able to use NMM to do both the summative ranking and leave some formative feedback.

    Hope NMM continues developing.

    Like

  4. I’m not sure anyone has pointed this out to you, but a Word document can be saved as a PDF. You could have the students do this very easily.

    Like

    1. Greatly appreciated. I have since learned this, but the experience (perhaps because they were Year 7s) taught me that it is far easier to collect in thirty pieces of work on paper, than it is digitally. In the digital sphere, there is so much room for work to disappear and excuses to arise. I have also found that asking for digital work usually prompts students to look things up on the internet which rarely ends well, without significant guidance. Students invariably copy or rewrite huge chunks of material which is sometimes relevant, but rarely makes sense to me, let alone to them!

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s