Blog·

Pairwise comparison beats individual scoring

An AI judge ranked 100 essays from the ELLIPSE dataset by comparing them in pairs. The ranking matched trained human raters on most pairs, and on nearly all where the raters saw a clear difference. Scoring each essay individually did much worse.

RankNameAgreement
  1. 01PairwiseClaude Haiku, 800 comparisons86%
  2. 02Individual scoringClaude Haiku, same rubric67%
  3. 03BaselineChance: a random order50%
How often each method put two essays in the same order as the human raters did, across 3,758 pairs.

Method

The test set is 100 essays from ELLIPSE, a public corpus of essays by English language learners in grades 8 to 12, all written on the same prompt. Two trained raters scored each essay for English proficiency on a five-level rubric. The average of their scores is the ground truth.

An AI judge (Claude Haiku) ranked the essays in two ways, with the same rubric:

  1. Pairwise. The judge reads two essays and picks the stronger one. 800 such verdicts are combined into one ranking.
  2. Individual scoring. The judge reads each essay alone and gives it a whole-number score from 1 to 5, the rubric’s scale, and the essays are ranked by score.

Judging criteria

The judge used ELLIPSE’s own five-level proficiency rubric, with every pair judged in both orders.

Judging criteria given to the AI

You are an experienced rater of English language proficiency in essays written by English language learners in grades 8 to 12. All essays answer the same prompt.

Judge which essay shows the higher overall English language proficiency, using the holistic scale below. Judge language proficiency, not the opinion taken or how persuasive it is.

5: Native-like facility in the use of language with syntactic variety, appropriate word choice and phrases; well-controlled text organization; precise use of grammar and conventions; rare language inaccuracies that do not impede communication.
4: Facility in the use of language with syntactic variety and range of words and phrases; controlled organization; accuracy in grammar and conventions; occasional language inaccuracies that rarely impede communication.
3: Facility limited to the use of common structures and generic vocabulary; organization generally controlled although connection sometimes absent or unsuccessful; errors in grammar and syntax and usage. Communication is impeded by language inaccuracies in some cases.
2: Inconsistent facility in sentence formation, word choice, and mechanics; organization partially developed but may be missing or unsuccessful. Communication impeded in many instances by language inaccuracies.
1: A limited range of familiar words or phrases loosely strung together; frequent errors in grammar (including syntax) and usage. Communication impeded in most cases by language inaccuracies.

Do not reward length for its own sake.

Each ranking is evaluated on the 3,758 essay pairs the human scores separate: the measure is the share of those pairs it puts in the same order as the humans. A random order scores 50%.

Results

The pairwise ranking put most pairs in the same order as the human raters. Its errors are concentrated in close calls: where the raters’ scores differed by 1.5 points or more, it agreed on 98% of pairs.

50%60%70%80%90%100%Close call0.5 ptClear1 ptVery clear1.5 ptObvious2+ ptPairwise, 0.5 points apart: 78% of pairsPairwise, 1 point apart: 89% of pairsPairwise, 1.5 points apart: 98% of pairsPairwise, 2 points apart: 98% of pairsPairwiseIndividual scoring, 0.5 points apart: 60% of pairsIndividual scoring, 1 point apart: 69% of pairsIndividual scoring, 1.5 points apart: 81% of pairsIndividual scoring, 2 points apart: 88% of pairsIndividual scoring
Share of essay pairs put in the same order as the human raters. The axis starts at 50%, the score of a random order.

Individual scoring by the same model ordered far fewer pairs correctly. It gave 78 of the 100 essays the same score, a 2, so most essays tied.

Human raters, average of two

1.522.533.544.5

AI, individual scores

12345
Number of essays at each score.

Pairwise agreement with the raters was comparable to the raters’ agreement with each other. Where both raters separated two essays, they agreed on the order 93% of the time, but they gave the same score to 30% of pairs. The pairwise ranking orders every pair.

Length does not explain the result. A ranking by word count is barely better than chance.

Example judgments

1 / 8
Agrees with the raters
Essay ARaters' pickAI's pick
4.0raters' score
out of 5
Going to school is the worst way to attend classes. Online or video conferencing would benefit students to attend classes. Although going to school helps. Taking a class at home, student don't have to worry about classmates bulling, being late cause there is no place to travel, and having no food to eat in school. Waking up early every morning to school can be very stressful. Students have to take a shower, brush their teeth, some girls put on makeup, and then packing the school bag. Students often get lazy getting up from bed therefor missing the bus. So they have to stay home, maybe wait for a ride, or even walk to school. Most students don't get a good amount of sleep that makes their focus off in class, also the hunger get the best of them which makes it worst. Working online students may pick the time that best fit them and the student stay home. So he or she don't need to worry about looking good. Attending online classes there will be no classmates which is a good thing. Having no classmates he or she can focus on work, because the student is not in the class with a lots of people that will makings noises or distract the student. Since your not in a classroom nobody can bully, tease, or judge the student. That makes some students go though depression in school by bullies. For example the way he or she dress, looks, smells, or did in the pasts. It will make students not attend school no more. Therefor online classes students will be working by their self which will give them more focus on school work. What can you benefit attending class from school? Going to school he or she can commutative with other people to make friends. Students can ask teacher or friends for more help. Cause the student is going out parents are saving money by the electric bill will be lower. By he or she going to school they can do sports or programs after school to help choose their career, and not be coup up staying at home. In conclusion attending online class can benefit the students. It the best way to get the education. By keeping the students safe from bullies, no distraction from classmates, always have food to full up the belly to focus, and taking online classes that can work on the students time.
406 words
vs
Essay B
2.5raters' score
out of 5
In may opinion, i think students they can't benefit by attending classes at home. Because it is going to be so mach difficult for they working at home stading or doing homework by them self and by taking a classes online or video conferencing is going to be mach difficult then taking a class with a real teacher and not on the computer. Is a lot of distraction at home like children's, noises, and animals so that is going to make a distraction for them so at home it is not going to be a good place for them to study, and student they can't concentrate when they are working so study at home is not going to be a place for student's, and some of the students they don't care about studding thats why they prefer to study at home were no one is going to tel them noting if he don't do homework or study for a test. Students they can get boring, tired, or angry when they don't no how to do sameting and they don't get help from nobody to do homework or something that they don't no how to do and then they star getting deprecation or headech. The best way the students can be benefit, is going to school, get help from teachers, learn more then study at home, they can do something different like do sports, playing music, or something that is going to make their future grad.
245 words

◆AI judge

Text A shows more varied syntax, longer and more developed sentences, and a clearer organizational structure with paragraphs covering distinct points and a conclusion, though it has frequent errors (e.g., 'bulling', 'therefor', 'the student stay home'). Text B has more pervasive errors in spelling, word choice, and mechanics ('mach', 'studding', 'noting', 'deprecation', 'headech'), with very short, loosely connected sentences and communication impeded more often. A is therefore the stronger language performance, though both fall in the lower bands.

Same pick with the essays in the other order.

Limitations

  • One sample of 100 essays, one prompt, one run.
  • ELLIPSE has been public since 2023, so the essays may be in the model’s training data.

Conclusion

Pairwise comparison was far more accurate than individual scoring, with the same model and the same rubric.

  • Fewer than half the errors. Pairwise put 14% of essay pairs in the wrong order, against 33% for individual scoring.
  • Near-perfect on clear differences. Where the raters’ scores differed by 1.5 points or more, pairwise agreed on 98% of pairs, against 83%.
  • As consistent as a second rater. Pairwise agreed with the raters about as well as they agreed with each other.
  • Individual scoring could not separate the essays. It gave 78 of the 100 essays the same score.
Full results

Pairs in the same order as the raters’ average. These are all 3,758 essay pairs the raters’ average separates. Pairs a ranking ties count as half right. Intervals are 90% bootstrap intervals over essays.

Ranking Same order
Pairwise 86% (81 to 90)
Individual scoring 67% (63 to 72)
Word count 57% (50 to 64)

By how far apart the raters’ average put the two essays.

Score difference Pairs Pairwise Individual scoring Word count
0.5 1,773 78% 60% 56%
1 1,259 89% 69% 56%
1.5 or more 726 98% 83% 62%

Against a single rater, compared with the other rater. Ties make this comparison depend on how they are counted, so both ways are shown.

Ranking Ties count as half (avg. of both raters) Only pairs both separate Share of pairs it separates
Rater 1 against rater 2 79% (73 to 83) 93% 70%
Pairwise 87% (82 to 91) 89% 100%
Individual scoring 69% (64 to 74) 93% 44%

Rank correlation (Spearman, ×100), the standard measure in essay-scoring research. Intervals are 90% bootstrap intervals.

Ranking Against one rater (avg. of both) Against the raters’ average Top-quarter overlap
Rater 1 against rater 2 61 (50 to 70) n/a 60%
Pairwise 68 (58 to 75) 75 (65 to 83) 68%
Individual scoring 51 (41 to 61) 55 (45 to 66) 52%
Word count 15 (0 to 30) 17 (0 to 34) 24%

Top-quarter overlap is the share of a ranking’s top 25 essays that are also in the humans’ top 25.

Differences in rank correlation (paired bootstrap, 90% intervals):

  • Pairwise minus rater against rater: +7 (−3 to +17).
  • Pairwise minus individual scoring, against the raters’ average: +20 (+9 to +31).

Length. Controlling for word count leaves the pairwise rank correlation at 75. The pairwise ranking correlates with length at 29, the humans’ at 17.

Method.

  • Sample: 100 essays drawn at random from the 192 in the test split on the prompt “Distance learning”.
  • Human scores: each rater gave a whole number from 1 to 5. The ground truth is their average, which runs from 1.5 to 4.5 with a mean of 3.15. The dataset has no human ranking. The human order comes from sorting by these scores, so many essays tie.
  • Rubric: ELLIPSE’s holistic level descriptions, verbatim, plus “judge language proficiency, not the opinion taken” and “do not reward length”.
  • Pairwise run: each essay met 8 opponents, and every pair was judged in both orders. The ranking is a Bradley-Terry fit of all 800 verdicts.
  • Individual scoring: one whole-number score per essay, from 1 to 5. Each human rater also gave whole numbers; the half points come from averaging two raters.

Judge checks.

  • The judge picked the essay it read first 64% of the time. Judging both orders cancels this out.
  • It gave the same verdict in both orders 61% of the time.
  • Split-half reliability was 0.71.

Caveats.

  • ELLIPSE excluded essays and raters its authors found unreliable, which raises the human benchmark.

Prior work. An EMNLP 2025 study found the same direction on another essay corpus: GPT-4o agreed more with human scores when comparing pairs than when scoring directly.

Rank your own stack.

The same judges, pointed at your documents.

Join the waitlist