Key takeaways
- EnlightenAI was within one point of the official score on 96% of essays, and 99% in ensemble mode.
- Its agreement score was 0.84. For its 2025 tests, New Jersey reports 0.72–0.91, mostly comparing its scoring engine with a human scorer.
- One sentence of calibration in the rubric's language raised exact matches by more than 7 points.
- The data: 207 officially scored essays across 15 released tasks, grades 3–10.
What we tested
New Jersey doesn't publish scored NJSLA student responses; it points teachers to PARCC's released items. NJSLA kept PARCC's task types and scoring traits, and PARCC released annotated, officially scored responses.
We used every released task with the full prompt, the passages and at least eight readable responses: 15 Literary Analysis, Research Simulation and Narrative Writing tasks, grades 3–10, 207 essays, scored on Reading Comprehension and Written Expression (0–3 in grade 3, 0–4 above). After five examples per task, 132 essays were graded.
How EnlightenAI compares with NJSLA scoring
New Jersey's scoring engine gives about 90% of first scores, and a second score goes to 10% of responses, usually from a human. The agreement New Jersey reports is a mix of engine-and-human and human-and-human pairs, and it doesn't publish human-only figures. So this is a looser yardstick than the other states'. How to read state scoring results explains the differences.
EnlightenAI was within one point of the official NJSLA score on 96% of essays, and 99% in ensemble mode.
Agreement score with the official score (0 to 1)
EnlightenAI, production
EnlightenAI, ensemble mode
Written Expression (for most tasks, the combined Reading Comprehension and Written Expression trait), grades 3–9, spring 2025. Their bar shows the middle of the reported range. Source: NJSLA Spring 2025 Technical Report, Table 4.6 (pp. 56–57).
One sentence of calibration
NJSLA scorers require every bullet of a score point, so a response between two levels gets the lower one. We added that rule as a single plain-language note, written on half of the tasks and checked on the other half.
| Setting | Essays | Exact match | Within one point | Agreement score (QWK) |
|---|---|---|---|---|
| Production, with the note | 132 | 56.8% | 96.2% | 0.84 |
| Ensemble, with the note | 132 | 54.5% | 99.2% | 0.86 |
| Ensemble, without the note | 132 | 47.0% | 95.5% | 0.81 |
On the half of tasks the note was never adjusted on, it raised exact matches from 44.7% to 51.3% and within-one-point agreement from 97.4% to 100%. This is the kind of guidance any teacher can add in EnlightenAI's special instructions.
How we ran the study
We ran every study the way a teacher uses EnlightenAI. For each prompt, five officially scored papers that span the score range act as the teacher's graded examples, with the state's written explanation as their feedback. EnlightenAI then grades every other paper for that prompt once.
- Examples always come from the same prompt. We never mix prompts, grade bands or languages.
- Papers used as examples are left out of the results, and no paper is counted twice.
- Prompts, passages and rubrics are used in full.
- Any calibration note was written on half of the prompts and checked on the other half.
The grader runs in EnlightenAI's production setup at production settings, with no fine-tuning and no second review pass. It writes rubric-based feedback first and then scores. We also report an optional ensemble mode, which writes nine drafts in one request and keeps the middle score on each trait, so regrades come back steadier.
A generous test for any grader
Released state papers are the right place to check that a grader applies the state's standard, but they are a generous test for any grader, including ours.
- They are the clearest examples. State scorers chose them because each one shows a score point cleanly. Everyday writing on a new task is messier.
- They are online. AI models may have seen them in training, which can inflate results.
- They are easy to tune to. Adjusting a grader while looking at the test papers inflates results, so we calibrate on half the prompts and report the other half.
That is why we also test on essays from the field with DREAM Charter Schools, which have never been online. There, EnlightenAI agreed closely with DREAM's official scores. For what each state's numbers mean, see How to read state scoring results.
Limits
- These are PARCC-era released papers, used because New Jersey hasn't released scored NJSLA responses. NJSLA uses the same task types and scoring traits.
- New Jersey's reported range mixes engine-and-human with human-and-human pairs; it doesn't publish human-only agreement.
- 132 graded essays is a modest sample. Exact-match figures carry a margin of roughly ±8 points.
Sources:
FAQs
Quadratic weighted kappa (QWK) is the standard way state testing programs measure how closely two graders agree. It runs from 0 to 1. It gives more credit for close misses than far ones, and a grader that gives every essay the same score gets 0, however often it happens to match. Above 0.6 is considered good and above 0.8 is excellent.
Yes. Each study has a verification kit with every graded response, the examples used, our scores and feedback, and a script that recomputes the tables. Get in touch to request one.
