Using Comparative Judgement

Some practical reflections on its use in practice

I was first made aware of Comparative Judgement as a method of assessment last year, through one of David Didau’s informative blogposts. I had always meant to get around to using it, but was put off by a fear of using technology. I have regularly compared scripts when awarding marks, and have on occasion sought to put together some sort of order before being brought back to the use of by my Deputy Head, and fellow A-level history teacher, to mark some Y12 mock essays.

Having had some new, functional photocopiers installed with a scanning function, I was willing to press ahead. I shall outline the process below for the uninitiated and then offer a simple evaluation of its value below. I’ll probe these thoughts more deeply later in the week.

The Process

  1. Scan in the exam scripts. Really easy if you have a ‘scan to USB’ function on your photocopiers. I’ve become a dab hand at this. You’ll want to use an easy code (like P12 for the twelfth student in 8P) to name the files, rather than perhaps typing in all of their names. Each essay/piece of work needs to be scanned separately. It took me about 15 minutes to scan in 46 sixth form mock scripts.
  2. Upload the scripts to a new task on which is free to use.
  3. Get judging. It took a Luddite such as myself a little while to find this function. Bizarrely, the web address to access the scripts is located in a section called ‘judges’ but once there you simply click left and right, depending on which script is better in your opinion. Nomoremarking recommends going with your gut and taking less than 30seconds to make a judgement. In practice, this was true of some Y8 essays I’ve compared, but sixth form essays took an average of three minutes to judge.
  4. The data coming in is easy to read. You are provided with a downloadable readout of the rank order of your pupils. It also comes with an ‘Infit’ score to consider which essays the software is less confident in placing. This is often where you have invited multiple judges, and you have perhaps implicitly disagreed on its value.
  5. Apply some marks. I have been less sure of this. However, I’ve read a selection of essays, found some on the level boundaries, applied marks, and then distributed the marks evenly throughout the levels.
    On essays where the Infit score is above 1.0 (indicating unreliable judgements) we’ve had some really interesting discussions about the merits of the essays, what we should be looking for and then manually awarded marks using an exam board mark scheme. I think it is clearly going to be valuable if you bank scripts from year to year with marks you are confident with, and feed them in – this should save time in awarding marks, if you have essays with firm marks already in the mix.

Dare I say it, judging essays has become fun. The clicking gamifies marking and I’m in a scramble to meet my marking quota. we have found that multiplying the number of scripts by 3 to determine the total number of judgements that need to be made, and evenly dividing this up between the team of markers works fine. In practice, essays are being compared against others 8 times there, and we’re achieving a reliability score of over 0.8 which David Didau says is the goal, and in excess of national examinations.

Strengths Weaknesses
Marks are awarded with great confidence, and a reliable set of data on the rank order of class is valuable for a range of accountability measures & considering further interventions. It is difficult to overcome the urge to write individualised comments on essays. Students (and SLT?) need to expect feedback where this isn’t the case. This feeds in with Christodoulou’s recent work on separating out formative & summative assessments.
It’s quick. Doesn’t sound like it, but marking those mocks could easily have consumed 8 days at 15-20 minutes an essay. At 3 minutes an essay, plus scanning, plus determining marks (30 mins when you have no known marks within the judgement process) is significantly quicker. Transforming electronically judged essays into generic feedback for pupils requires careful thought. I’m still refining this.
There is less of a marking bias. Especially if you ask pupils to submit essays with a code (see part one above) rather than naming them. Essays that ‘miss the wash’ are troublesome to reliably fit in the process. This is probably more frustrating rather than the end of the world.
I have thought much more carefully about what I’m really looking for in essays. I think this has led me to be clearer, already, with my classes about how they need to develop their essays. Getting an entire team on board with this might be more difficult than using the software individually. If you’re marking procedure is out of step with other staff, as a head of department, you can still have little confidence in the reliability of marks generated.


How I intend to develop my use of comparative judgement further

  • Ask students to highlight key areas of the script. This might involve showing the mark scheme, and asking them to pick out the five sentences they most want the examiner to see. This should speed up comparisons. Before I stuck my first essays through the process, I had already put formative comments on them. These were a useful aid in passing comment.
  • Banking essays for next year with secure marks attached to them. This should eliminate significant amounts of time transforming the rank order into marks.
  • Get students to submit work electronically. I am in the midst of getting KS3 to do this with an outcome task to a unit of work. I’m not sure how valuable this will be. Paper, pen and scanning seems to be less hassle, so far.**
  • Learn what this anchoring business is which seems to be taking comparative judgement to the next level by connecting subsequent pieces of work together. If I get to the bottom of this, I’ll blog on it.


Comparative judgement seems to be a valuable tool for making summative judgements on the quality of pupils’ work. It does not replace marking, or feedback but these should be steps on the road towards a final piece of work. This is where the comparative judgement bit fits in.

Your thoughts and questions are invited.


** Update – I have now discovered that will not accept word documents. They need to be PDF files which throws this plan of mine out of the window.


Why should we share the work of academic historians?

Rachel Foster’s attendance at the WLFS history conference stirred a rather interesting discussion within my own department about the role of academic history in the classroom. Inspired by Foster’s talk, her excellent chapter in Debates in History Teaching, discussions with fellow participants of my MA and our own fertile minds we devised a list of why using academic history in the classroom is valuable. In no particular order, we suggested:

  • To provide the narrative. Historians can compel the interest of students in ways that perhaps we cannot.
  • To provide competing interpretations.*
    • To identify the key debates in history.
    • To explore how historical interpretations are constructed.
  • To model styles of writing, which was the basis of Jim Carroll’s workshop at the WLFS conference.
  • To develop historical reasoning.
  • To judge pieces of academic work. Arthur Chapman has shown me a number of examples where historians have willingly engaged with pupils in debate and assessed their work.
    • To enthuse and motivate students. Diana Laffin has a book club with her sixth formers.


There is an entirely different debate to be had, which I will blog on, regarding the limitations of using historical scholarship, and reducing our subject to academic history. History, of course, extends beyond academia.

I am keen to discuss the ‘how’ and ‘why’ use historical scholarship further. But the more important question remains the what. It is perhaps overly optimistic to suggest that more traditional schools, with their claims of quicker lesson planning, and improved behaviour management allows teachers the room to appropriately develop their subject knowledge. I’m perhaps projecting my own shortcomings onto the broader community. But given the age profile of participants in the WLFS history conference, I’d suggest I’m in good company when I say that I’m not on top of all of the historical debates surrounding the topics that we teach in school. There are some killer passages of text out there. Christine used a now well used extract from Simon Schama, “coffee table history” to illustrate a clear argument and style of historical writing. I am most particularly pleased with a section of text that I use from A Concise History of Australia to illuminate for my year nine students what life was like for convicts in Australia in the mid-nineteenth century.

There are a range of passages of text just like this. Those that you get an instinctive feel when you’re reading them that they offer you something special. One sentence gets write to the crux of the argument. A turn of phrase that beautifully ties off an entire monograph’s worth of argument. But they are in those entire monographs. One of the strengths of the history teaching community is its size, and its passion for its subject. I have yet to meet two history teachers with identical interests in specialisms. It strikes me that we need to do more to share those ‘killer passages’ when we see them. Those that we can share with students of year seven, or at least aspire to build them up towards. It is laudable the West London Free School sets a piece of academic reading each fortnight. They follow in a fine tradition of history teachers bringing historians into the classroom. However it speaks volumes that the Historical Association does not have a page to draw together these uses of academic history, in the way that it does collate thinking on a range of other curriculum issues.

History teachers are instinctively a sharing bunch. This is clearly seen with the exchange of resources and ideas following conferences such as SHP, and Robert Peal’s online resource collection to accompany his textbook series. We need to begin sharing the way we use historians in the classroom in the way that we do other resources. It would be valuable to crowdsource ways that we facilitate this process. Would one teacher and a dropbox account suffice or do we need a bigger vision?


*Ben Walsh gave one of the best lines of the WLFS conference, in suggesting that we do not send twelve-year-olds to fight grizzly bears. We send another grizzly bear. Students should not be evaluating historians’ interpretations but instead seeing how their views differ. This builds on Counsell’s own suggestions that students should ‘hear the shape and style’ of historians’ arguments.

Where can I read about the WLFS History Conference?


  1. The Twitter hashtag #WLFSHistory
  2. My Blog!
    1. My review of Christine Counsell’s opener.
    2. My review of the WLFS approach and Jim Carroll’s session.
    3. My review of Vartan Tamizian’s session on historical narratives, and Robert Tombs’ closing lecture.
  3. Other valuable blogs on the WLFS Conference:

    1. Conference organiser extraordinaire Louis Everett has posted his own reflections and the resources for the day.
    2. Lee Donaghy has an exceptional reflection here, which is valuable in picking apart some of the individual workshops and the ‘revolutionary’ idea from Michael Fordham that the curriculum is the progression model.
    3. Robert Peal’s discussion on the importance of knowledge, and how this message manifested itself through the conference.
    4. Alex Ford carries on the conversation about what we want from textbooks.
    5. A further set of reflections by Miss Beckwith.
    6. Sally Thorne writes some captivating reviews of the various workshops in a series of blogposts here.
    7. Michael Fordham has continued to explore how we might conceptualise progression in history.
    8. Tom Rogers has a very interesting perspective on the ‘big picture’ knowledge debate in his TES column.

WLFS History Conference – Part Three

Narratives are Complex

You can find a guide to my three posts, and other people’s reviews here.

The session that I took most from was probably Vartan Tamizian’s session on historical narratives. Unfortunately, my journey here started from the “narrative analysis” questions in the Edexcel GCSEs. However Vartan very usefully fused my thinking, which to this point had remained compartmentalised into separate silos in my head: historical frameworks, which I have been working on for my MA, and narrative accounts in history. This, I think, demonstrates the value of these conferences. One of my colleagues in RS perhaps puts it best, by suggesting that there are no new ideas in teaching, but it can be valuable to be reminded of them.

Vartan’s session was typical of the day, in that it had both strong connections with existing historical discourse, with a wealth of references to literature-review level research and to academic history. I was hooked from the outset. Participants were given a wealth of examples of how narratives are used across the curriculum. One idea I particularly liked was at GCSE, where students produce a “living document”, a narrative of their course so far that they review and update in each school holiday. These narratives serve as excellent revision tools, as well as being quick diagnostic tools for the teacher. Narratives can be used to embed knowledge; the Michaela school have said much on the power of ‘stories’ to develop memory.

In many ways, Vartan was perhaps preaching to the converted. The real value in the session came in how to construct historical narratives. I look forward to trying these ideas out in the classroom, but the clear distinction drawn between actors in history, and external events (“happenings”) seems an obvious and valuable way of breaking down the process of generating narratives.

Robert Tombs

Narratives are inherently complex. One of the challenges of delivering narratives to students is that we need to problematise them. History is too readily abused. Students too easily seize on parts of narratives that fit their existing world-view, their prejudices and discard the rest if they are not taught how to challenge the narrative (Howson 2009). This would involve the clear teaching of different historical interpetations. In light of this, I have wrestled with some elements of Robert Tombs’ closing lecture. It seems to have been an exceptionally popular part of the day, so I write this with some trepidation. But the idea that teaching an English narrative of history struck me as somewhat troubling. Tombs was very deliberate in drawing on the rise of Mary Seacole in history, as a cautionary tale and an example of why we should perhaps return to a more traditional curriculum. It was not clear that the Seacole myth has already been thoroughly debunked. I wasn’t always entirely clear on how far Tombs was echoing Counsell in the morning session. Counsell had passionately reminded us that “to challenge the canon, we must teach the canon” (Christine clearly read my most recent MA essay…!). Was this Tombs’ message too? I felt that it was, on balance, but the very deliberate point of excluding new minority histories was perhaps bristled within the more nuanced points that were being communicated.


I would like to think that the narrative of the day was one of knowledge, and providing the necessary disciplinary knowledge to make it powerful. We can all agree that the day revolved around a committed group of history teachers, wrestling with the complexity of their subject, facilitated by an excellent subject team at the West London Free School.


Roll on next year’s conference Louis! And St. Totteringham’s Day


Howson, J, ‘Potential and pitfalls in teaching ‘big pictures’ of the past’, Teaching History, 136, (2009).


WLFS Conference – Part Two

The WLFS approach to the history curriculum

Part One can be found here.

It was a pleasure to attend a session led by the head of department, Louis Everett. His session was an excellent blend of confidence in his department, but humility to invite collaboration and critique from colleagues. The session invites a few questions, but perhaps offers a template for departments to follow more broadly.

Louis sees the purpose of the KS3 curriculum as providing the residual knowledge that would be needed at KS4. Indeed, he mentioned several times the themes, and the links that he teases out across the 11-18 curriculum. This very much built on Counsell’s reminded that first-order concepts like parliament, heresy, the church etc are reinforced by being revisited in a number of different contexts.

Louis has also led his department to take quite a strong stance on feedback. Individual comments are not written on pupils’ work. Instead, pupils work is given codes which relate to comments that are delivered as part of whole class feedback. This strikes me as sensible and an approach to feedback that is spreading. I’m not yet ready to abandon individual written feedback yet though; it has a time and a place and I wasn’t sure that it had been entirely dispensed with at the WLFS.

Attached to this, both Louis and his colleague, Vartan Tamizian, referred to the wonders of a piece of software called ‘Show My Homework’, something I’ve made a note to look into further. I’d be grateful anybody can elaborate on the virtues, or indeed the pitfalls, of this.

The other message emerging from Robert Peel and Louis’ sessions was the rigid structures relating to homework. Homework tasks alternate week by week, where fortnightly students are set a ‘self-quizzing’ homework from their knowledge organisers and in the other week they read a piece of academic history. I’m keen to unpack this a little more.

Firstly, I like the use of knowledge organisers. Christine had rightly raised a few questions about their use in her morning lecture and I have blogged about their complexity previously. We cannot rely on them as pupils having genuine historical knowledge, and more research needs to be done into how they can be most effectively support the acquisition of historical knowledge. However the strong routines, high expectations and clear guidance on the knowledge required an only be good for pupils.

In terms of using historical scholarship, I’m keen to learn more about the WLFS’ approach. Rachel Foster also ran a session on this, which I may blog on at a later date when I have discussed with my colleague what he was able to extract from the session. I imagine her work was very similar to her chapter in Debates in History Teaching. Louis rightly mentioned the importance of helping students to make sense of this literature and I hope the WLFS engages with teachers to share their sterling work in this area. I wasn’t convinced by the suggestion that centralised detentions freed up enough time for staff to read sufficient academic history and maintain their subject knowledge. I have mentioned previously, on Twitter, how important it is that we share accessible, rich and valuable sections of text as a history teaching community to save us the time of researching material further.

I particularly enjoyed Jim Carroll’s session, which gave us one idea on how we might use historians’ writing as a model to aspire to with our sixth form students. His approach is an interesting one, which can be read about more fully in Teaching History. One participant in the session did question the virtues of writing with a more dense, complex style. This was a valid question, and teachers need to think through carefully precisely which aspects of pupils’ writing we might want to develop. Jim’s point was a clear one though, that we can improve pupil’s history by improving their historical writing and there is a growing body of literature out there to help teachers with this.

I aim to follow up with one, possibly two, further posts to review the other workshops I attended.

WLFS Conference – Part One

Knowledge is not an end in itself

The West London Free School history conference was an excellent opportunity to discuss a knowledge-rich approach to history teaching. I have often valued a truism presented by Mike Hughes to an INSET session I attended in my NQT year, that any task is only as good as the quality of the dialogue it provokes. It is with this in mind that one should think about this conference. While I may evaluate the ideas presented in a series of blog posts, the conference as a whole was excellent in promoting discussions between history colleagues, expanding the horizons of many of them. Lunchtime* was characterised by colleagues discussing what they could, should and would take back with them to their own departments.

The day started with a curious introduction from head teacher Hywel Jones. It is always good to see a senior leadership team supporting staff in their extracurricular endeavours and it was clear from the outset that there is a clear philosophy engrained throughout the school. The message was a strong one, that knowledge was valued in this school and passing on a knowledge-rich curriculum was vital to students’ success. It was for Christine Counsell to introduce a bit of nuance on the precise role of knowledge and importance of knowledge in the curriculum.

I do not intend to summarise the contents of Christine’s speech in full. I will, however, point out the key messages that I took away.

  • Christine had an important point to make about humility. I think this is important in the context of the current pedagogical debates taking place through blogs and on Twitter. Many straw men are being set up, heinous schools out there which deny students’ knowledge. I am not sure how far that is true, not in 2017 at least. Concrete evidence would be welcome. Certain commentators could take heed of this note of humility.
  • Assessment theory has been absent thus far from recent debates on knowledge. So far as it has been discussed, it has been in a limited way and brought back into fashion low-stakes testing. I have always been keen to remind my own students, particularly those preparing for external assessments, that those tests take a sample of students’ domain knowledge which needs to be built up over time. The upshot is that we need to be laying the foundations of knowledge, and the tools for manipulating it, over time.
    • Christine helpfully reminded us of ‘timeline tests’ where students need to plot their knowledge, in chronological order, to reach a threshold standard before taking a summative test. These ideas are not new, but a valuable reminder. Whilst submerged in curriculum reform at KS4 and KS5 it is easy to forget such basics.
  • This speech also helpfully distilled recent cognitive psychology on knowledge and its role in learning. Students need ‘fingertip’ knowledge to help support more advanced, second-order thinking (a theme that was not as prevalent during the day as it could have been). Once students have used and deployed that knowledge, they are left with “residue knowledge”. Recent work on knowledge organisers, I think, has taken quite a short-termist approach and my sense was that the WLFS put more emphasis on what “residue knowledge” we want students to have in five years and perhaps ten years after their schooling. It was at this point in Christine’s speech that I reflected on Fordham’s post on what knowledge is cumulatively sufficient; what is the relationship between the little details and the big picture? As a history community, our debates could be infused with more of this thinking and language.
  • The expression “cumulative assessment” was used, but not really elaborated upon. This too is perhaps important. I would contend that we want students’ residual knowledge to be a broad, overarching framework of the past. Students should be able to see the broad arcs of change in history, and analyse those. Not only were frameworks of knowledge** not discussed, but cumulative assessments seem as though they might be a useful step towards constructing them.


* I live my life through my stomach, and thoroughly enjoyed the lunch offering! An excellent effort by the school’s catering team.

**I use the expression in a Shemiltian sense. See Nick Dennis’ blog for a brief introduction to frameworks of the past.

Evaluating the first meeting of the research group

This post follows on from this article on ‘planning the first meeting of a research group.’

My first session was initially designed to gauge the levels of interest, experience and expectations of colleagues that decided to attend. From my perspective, the group felt a little rudderless but certainly achieved this aim. Helpful colleagues of mine have also pointed out that the group perhaps should not necessarily have a strong direction but instead it should guide colleagues to explore research for themselves – something to think about a little bit more fully when planning the next session.

Colleagues were very interested in research, and saw it more as an opportunity to ‘learn something new’ which I found quite interesting. I’m not sure how far educational research can achieve that, or how far this chimes with the aims of the group. Ideally, I want colleagues to question their teaching practice and use evidence to guide changes. If this involves learning something new, then great, but I’m not sure they will receive the silver bullet they’re looking for. It is perhaps worth noting, at this point, that the session fell in the week that we picked over the bones of our mock results. Staff were naturally looking for “interventions” that would turn around students who were not performing at the expected level.


Reflections on own performance:

This group sits as part of my own personal development. I have been guided by the senior leadership team this year to think carefully about how I might grow into a successful leadership role. This has therefore led to the following thoughts:

  1. Staff respond well to experience. I am not the most experienced teacher, and most of my participants were more senior to me. However they responded well to my background as an MA student currently engaged in the research process myself: this was something I was perhaps anxious about before the session.
  2. Staff are supportive, and are generally willing to be led to where you want them to go. In hindsight, I wished I had used this session to unpack more fully what colleagues meant by wanting more research on successful interventions. They can only make progress in this area if they have more concrete ideas about what aspects of their student’s learning they wish to develop.

Reflections ahead of the next session:

  1. Staff want a range of ideas. I am therefore minded to provide them with a few different articles, summarised, and invite them to discuss which of them might prove the most useful to them in designing an intervention to improve a set group of students in their exam classes. This should hopefully encourage them to think in a more focussed way, and will allow us to plan some useful interventions to be evaluated in a subsequent session.
  2. I want to send staff off with a concrete task to complete. This session didn’t have the ‘zinger’ of an ending I had hoped it would – largely because we were all speaking about attending the next session, which does not fall until the beginning of May. This should take the form of a structured, planned use of some research to modify (or perhaps even remove) an element of their current practice.

Reflections on the role of the group next year:

  1. I have mentioned this elsewhere, but the group does need to meet more regularly. I need to have, by the end of this academic year, put together a clear proposal for the model the group should follow in 2017/18.
  2. With a clear programme for 2017/18 staff should be encouraged to commit to the group for the year and perhaps even be invited to join. This allows for the action-research model I am aspiring towards to be sold more clearly before the first session of this year. The first meeting of the group did feel like colleagues were arriving with broad, differing expectations. The session had also come as something of an interruption to their year-long professional development programme which has, of course, been underway since February.


I remain interested in speaking with fellow research leads and others bringing research into the classroom. Please do get in touch, I have always got the time to chew the cud on this subject.