Key takeaways
- EnlightenAI's agreement score on Organization/Purpose was 0.81. On 2025 test essays, a first scorer and an expert second scorer agreed at 0.77.
- It gave the exact official Conventions score 71% of the time; the two scorers matched 68% of the time.
- It was within one point of the official score on 97–99% of essays on every trait.
- Five scored examples made the difference: the same model with no examples reached 0.74; with five, 0.85.
What we tested
Smarter Balanced publishes released performance tasks with student essays, the official score on each trait and a note explaining it. Smarter Balanced is used in California (CAASPP) and other member states.
We collected all 20 tasks with scored essays: narrative, opinion, argumentative, informational and explanatory writing in grades 3–8 and 11, 220 essays in all. After five examples per task, 120 essays were graded.
How EnlightenAI compares with trained scorers
California's technical report shows how closely a first scorer and an expert second scorer agreed on about 95,600 essays from the 2025 tests. We measured EnlightenAI against the official score on released essays, which are clearer than everyday essays. How to read state scoring results explains why that matters.
On Organization/Purpose, EnlightenAI's agreement with the official scores was 0.81, in line with the 0.77 California reports for its own scorers.
EnlightenAI against Smarter Balanced scorers
Organization/Purpose: agreement score
Conventions: exact match
Scorers: weighted average of about 95,600 double-scored essays, grades 3–8 and 11, spring 2025. Source: CAASPP Smarter Balanced 2024–25 Technical Report, Appendix 8.I, Tables 8.I.15–8.I.21.
Full results
| Trait | Essays | Exact match | Within one point | Agreement score (QWK) | Two scorers |
|---|---|---|---|---|---|
| Organization/Purpose (1–4) | 120 | 67% | 98% | 0.81 (0.83 ensemble) | 71.4% · 0.77 |
| Evidence/Elaboration (1–4) | 120 | 62% | 97% | 0.75 (0.80 ensemble) | 71.2% · 0.77 |
| Conventions (0–2) | 120 | 71% | 99% | 0.59 (0.66 ensemble) | 67.6% · 0.66 |
The Evidence row combines Evidence/Elaboration and, for narrative tasks, Development/Elaboration, as California's tables do. California's second reads come from expert scorers, and essays where the two scores weren't adjacent were rescored before the statistics were computed, which raises the reported agreement.
How we ran the study
We ran every study the way a teacher uses EnlightenAI. For each prompt, five officially scored papers that span the score range act as the teacher's graded examples, with the state's written explanation as their feedback. EnlightenAI then grades every other paper for that prompt once.
- Examples always come from the same prompt. We never mix prompts, grade bands or languages.
- Papers used as examples are left out of the results, and no paper is counted twice.
- Prompts, passages and rubrics are used in full.
- Any calibration note was written on half of the prompts and checked on the other half.
The grader runs in EnlightenAI's production setup at production settings, with no fine-tuning and no second review pass. It writes rubric-based feedback first and then scores. We also report an optional ensemble mode, which writes nine drafts in one request and keeps the middle score on each trait, so regrades come back steadier.
A generous test for any grader
Released state papers are the right place to check that a grader applies the state's standard, but they are a generous test for any grader, including ours.
- They are the clearest examples. State scorers chose them because each one shows a score point cleanly. Everyday writing on a new task is messier.
- They are online. AI models may have seen them in training, which can inflate results.
- They are easy to tune to. Adjusting a grader while looking at the test papers inflates results, so we calibrate on half the prompts and report the other half.
That is why we also test on essays from the field with DREAM Charter Schools, which have never been online. There, EnlightenAI agreed closely with DREAM's official scores. For what each state's numbers mean, see How to read state scoring results.
Limits
- For two tasks, the released essays come from a pilot in which students saw slightly different sources than the current task.
- 120 graded essays is a modest sample. Trait-level figures carry a margin of roughly ±8 points.
Sources:
FAQs
Quadratic weighted kappa (QWK) is the standard way state testing programs measure how closely two graders agree. It runs from 0 to 1. It gives more credit for close misses than far ones, and a grader that gives every essay the same score gets 0, however often it happens to match. Above 0.6 is considered good and above 0.8 is excellent.
Two trained scorers reading the same essay don't always agree. States publish how often they do, and that is the fairest bar for any grader: if EnlightenAI agrees with the official score as often as two of the state's own scorers agree with each other, it is grading to the state's standard.
Yes. Each study has a verification kit with every graded response, the examples used, our scores and feedback, and a script that recomputes the tables. Get in touch to request one.
