Key takeaways
- On 147 grade 8 essays from DREAM Charter Schools, EnlightenAI matched DREAM's official score 70% of the time. A second experienced grader matched 47%, a reminder of how hard consistent essay scoring is.
- EnlightenAI was never more than one point off. Its agreement score was 0.77.
- These essays have never been online, so no AI model could have seen them in training.
- Rerun today with five different sets of examples, EnlightenAI averaged 0.72 and was within one point on 99.7% of essays.
Why essays from the field matter
Every state-test result in our research comes from papers the state published online. They are the right place to check that a grader applies the state's standard, but AI models may have seen them during training.
So we also test on student writing that has never been public. DREAM Charter Schools in New York City shared 147 grade 8 essays from one of its interim assessments, each scored 0–4 by a DREAM administrator on New York State's rubric. An experienced second grader also scored every essay independently, to show how closely two careful humans agree.
How EnlightenAI compares
We compared EnlightenAI and the second grader with DREAM's official scores on the same 147 essays.
On essays no AI has seen, EnlightenAI's agreement with DREAM's official scores was 0.77, and it was never more than a point off.
Agreement with DREAM's official score
Exact match
Within one point
Agreement score (0 to 1)
EnlightenAI used five of DREAM's scored essays as examples. Scores given, 0/1/2/3/4: DREAM 2/6/22/97/20; EnlightenAI 2/5/28/70/42.
Read the full story of this study in AI vs. human grading: why training is the game-changer.
Why exact match isn't enough
Two-thirds of these essays earned a 3. A grader that wrote “3” on every essay would match 66% of the time and land within one point 95% of the time, while missing every 0, 1, 2 and 4. Its agreement score would be 0.
That is why every study reports the agreement score (QWK) alongside exact matches. It rewards putting essays in the right order, from weakest to strongest, which is what students and teachers need from a score.
How we ran the study
We ran every study the way a teacher uses EnlightenAI. For each prompt, five officially scored papers that span the score range act as the teacher's graded examples, with the state's written explanation as their feedback. EnlightenAI then grades every other paper for that prompt once.
- Examples always come from the same prompt. We never mix prompts, grade bands or languages.
- Papers used as examples are left out of the results, and no paper is counted twice.
- Prompts, passages and rubrics are used in full.
- Any calibration note was written on half of the prompts and checked on the other half.
The grader runs in EnlightenAI's production setup at production settings, with no fine-tuning and no second review pass. It writes rubric-based feedback first and then scores. We also report an optional ensemble mode, which writes nine drafts in one request and keeps the middle score on each trait, so regrades come back steadier.
FAQs
Quadratic weighted kappa (QWK) is the standard way state testing programs measure how closely two graders agree. It runs from 0 to 1. It gives more credit for close misses than far ones, and a grader that gives every essay the same score gets 0, however often it happens to match. Above 0.6 is considered good and above 0.8 is excellent.
Yes. Each study has a verification kit with every graded response, the examples used, our scores and feedback, and a script that recomputes the tables. Get in touch to request one.
