Key takeaways
Grading a stack of essays consistently is hard, even for two experienced teachers using the same rubric.
In a DREAM Charter Schools study, two human graders scoring the same 147 essays only reached a QWK of 0.52, moderate agreement at best.
EnlightenAI reached 0.75 QWK against the DREAM administrator, and 0.68 QWK against a second teacher it had never seen grade before.
The takeaway: training matters more than raw technology. A teaching assistant calibrated on your rubric closes a consistency gap that generic AI grading can't touch.
Right now, across New York City classrooms, thousands of educators are racing against time. With state exams approaching, teachers are buried under mountains of student essays. Each piece will ideally be graded, returned, revised, and regraded, a cycle used for writing assessment across the country, from state exams to AP free response questions.
Behind closed classroom doors after the final bell, teachers struggle to give consistent scores and meaningful feedback on dozens, sometimes hundreds, of student papers. It's the educational equivalent of running a marathon every week.
This relentless cycle sparked two interesting questions for us:
In these real-world conditions, how reliably are we evaluating student writing? Could AI actually provide more consistent scoring than seasoned human graders?
To ascertain the consistency between AI and human graders, we first designed a straightforward experiment using EnlightenAI, the grading and feedback tool we developed within the Teaching Lab Studio. We partnered with DREAM Charter Schools in New York and analyzed 147 eighth-grade English Language Arts responses, real student writing from real classrooms.
These essays had already been graded by a DREAM administrator using New York State's 5-point rubric (0 to 4) during their regular assessment process. We simply wanted to see how closely EnlightenAI's scoring would match the human grader's holistic scoring.
To measure agreement, we used Quadratic Weighted Kappa (QWK), a statistical method which accounts for the alignment between two graders. A score of 1.0 indicates perfect agreement, with anything above 0.6 considered good and above 0.8 exceptional. Think of it as a rating of the consistency between two graders scoring the same student work.
The results? In our latest testing, EnlightenAI achieved a QWK of 0.75 when compared to the DREAM administrator's scoring, remarkably high alignment, consistent with or even exceeding typical benchmarks for automated scoring systems.
Put more simply, 100 percent of EnlightenAI's scores were within one point of the human grader's, and over 70 percent were an exact match. That's the kind of first pass a teacher can review and approve in seconds with an AI essay grader instead of rebuilding scores from scratch. |
But here's where things got interesting.
We wondered: across thousands of New York City classrooms, how consistently are different teachers scoring the same type of student work? To investigate, we recruited another experienced NYC teacher (unaffiliated with DREAM) to grade the exact same set of essays mentioned above, using the same scoring guide.
We expected strong alignment between these two seasoned educators. After all, they were evaluating the same student writing using the same criteria and rubric.
The result surprised us. The agreement between these two human graders reached a QWK of 0.52, indicating moderate agreement. Meanwhile, EnlightenAI, when calibrated with just five scored samples from the NYC teacher, achieved a substantially higher 0.68 QWK with that same teacher.
In other words, EnlightenAI was more consistent with each human grader than the two humans were with each other. Why? Our analysis indicates that there is a statistically significant effect when AI is calibrated to a consistent scorer. This points to the potential limits of general-purpose models and the importance of embedding human expertise into the grading process, even when AI is involved. |
This chart compares Quadratic Weighted Kappa (QWK) scores across three conditions. The results suggest that AI trained on a specific scorer's logic can produce more consistent results than even experienced educators grading using the same criteria.
EnlightenAI helps teachers deliver instant, rubric-aligned AI writing feedback so students can practice, revise, and improve faster. It's a simple way to start grading essays more efficiently.
These experiments point to something educators have long suspected but rarely been able to quantify: achieving consistency in grading student writing is remarkably difficult, even among experienced professionals using the same rubric.
The data suggests this isn't due to lack of expertise or commitment. Rather, human judgment in assessment is influenced by numerous variables, from fatigue to grading order.
Recall a famous study in which judges' willingness to grant parole dropped sharply just before lunch and rebounded after eating. It's a powerful reminder that even serious professional judgments are vulnerable to cognitive strain. Similar factors likely affect writing assessment. The sheer volume of writing samples requires educators to make thousands of difficult judgments annually, often under significant time constraints.
The implications are profound. When a student in the Bronx receives a score on a practice free-response question, that score, and by extension, the expectations of the student, might differ significantly if graded by a different teacher in Brooklyn or Manhattan. |
This inconsistency means we lack a reliable "true north" for student writing performance, making it difficult to accurately track growth over a school year. Factor in how immensely time-consuming it is to grade student writing, and it's no surprise schools struggle to build systems around it.
TNTP's seminal study, The Opportunity Myth, found that one of the strongest predictors of student achievement is consistent exposure to high-quality tasks and being held to grade-level expectations. Gaps in rigor and expectations help explain the persistent achievement divide between wealthy and under-resourced communities.
Our research suggests this challenge may be especially pronounced in writing, where the difficulty of consistent assessment makes it harder to define, let alone uphold, rigorous, grade-level expectations.
Writing isn't just about the final product, it's how students develop and demonstrate their thinking. |
But, until now, writing has been one of the hardest areas to assess accurately at scale. This need not be the case. With the right training process, AI can closely replicate expert human judgment, enabling a level of scoring consistency and feedback quality that has historically been difficult to achieve in the field. More importantly, it opens the door to codifying and sharing expert judgment in ways that weren't previously possible. AI calibration can allow an expert's thinking to be reflected well beyond their own classroom.
Better scoring and feedback is the key to demystifying the bar for student writing, and to make that bar transparent, consistent, and accessible across schools. At the same time, we can relieve the bottleneck that's long plagued writing instruction: the time, effort, and variability involved in giving high-quality feedback.
In partnership with leading institutions across the country, including districts using our AP English Language essay grader, STAAR essay grader, and CAASPP essay grader, we're actively researching how to effectively combine the benefits of AI with the indispensable role of the classroom teacher in the assessment process, and what this could mean for the future of writing instruction.
For readers interested in deeper analysis, we're currently preparing a research paper exploring these initial findings in greater detail. If you're curious what this calibration process looks like day to day, see how teachers generate anchor writing samples to calibrate a rubric or read our walkthrough on how to grade essays faster with a rubric-trained TA.
If you are interested in partnering or learning more, please email gautam@enlightenme.ai.
How consistent is writing feedback across your own classrooms right now? EnlightenAI learns from the way you already grade, so every score reflects your rubric instead of a generic AI standard.
