Key takeaways
- On Florida's anchor papers, EnlightenAI's agreement scores were 0.82–0.84 across all three domains.
- On everyday field-test essays, Florida reports 0.60–0.74 for two of its trained scorers and 0.67–0.80 for its own scoring engine. Anchor papers are clearer than everyday essays, so these aren't a like-for-like comparison.
- It gave the exact official score 72% of the time on every domain and was within one point on 99–100% of essays.
- No Florida-specific instructions were added. EnlightenAI worked from the rubric and five scored examples per prompt.
What we tested
FDOE publishes a B.E.S.T. Writing anchor set for each operational prompt: 16 student essays with official scores on Purpose/Structure, Development and Language (1–4 each), and a note explaining each score.
We used every recent set with readable essay text, including the Spring 2025 sets: 13 prompts and 159 essays, argumentative and expository, grades 4–10, with the current rubric for each grade band. After five examples per prompt, 94 essays were graded.
How EnlightenAI compares with Florida's scorers
Florida's technical report evaluates its own automated scoring engine. As a baseline, it shows how closely two trained scorers agreed on spring 2023 field-test essays, and the engine scored higher than that baseline. We measured EnlightenAI against the official score on anchor papers. How to read state scoring results explains why those two kinds of essays aren't directly comparable.
On Florida's anchor papers, EnlightenAI's agreement with the official scores was 0.82–0.84 on every domain.
Agreement score with the official score (0 to 1)
Purpose/Structure
Development
Language
Agreement score is quadratic weighted kappa (QWK). Florida scorers: two independent reads of 2023 field-test essays, grades 4–10. Florida's own engine reached 0.67–0.80 on the same essays. Their bar shows the middle of the reported range. Source: FDOE 2024–2025 Technical Report, Volume 4, Table 30 (pp. 36–37).
Full results
| Domain (1–4) | Essays | Exact match | Within one point | Agreement score (QWK) | Two Florida scorers |
|---|---|---|---|---|---|
| Purpose/Structure | 94 | 72% | 100% | 0.84 | 67.7–75.2% · 0.62–0.74 |
| Development | 94 | 72% | 99% | 0.82 | 66.5–74.1% · 0.60–0.73 |
| Language | 94 | 72% | 100% | 0.83 | 65.9–72.1% · 0.62–0.70 |
In ensemble mode, agreement scores were 0.84, 0.85 and 0.82.
How we ran the study
We ran every study the way a teacher uses EnlightenAI. For each prompt, five officially scored papers that span the score range act as the teacher's graded examples, with the state's written explanation as their feedback. EnlightenAI then grades every other paper for that prompt once.
- Examples always come from the same prompt. We never mix prompts, grade bands or languages.
- Papers used as examples are left out of the results, and no paper is counted twice.
- Prompts, passages and rubrics are used in full.
- Any calibration note was written on half of the prompts and checked on the other half.
The grader runs in EnlightenAI's production setup at production settings, with no fine-tuning and no second review pass. It writes rubric-based feedback first and then scores. We also report an optional ensemble mode, which writes nine drafts in one request and keeps the middle score on each trait, so regrades come back steadier.
A generous test for any grader
Released state papers are the right place to check that a grader applies the state's standard, but they are a generous test for any grader, including ours.
- They are the clearest examples. State scorers chose them because each one shows a score point cleanly. Everyday writing on a new task is messier.
- They are online. AI models may have seen them in training, which can inflate results.
- They are easy to tune to. Adjusting a grader while looking at the test papers inflates results, so we calibrate on half the prompts and report the other half.
That is why we also test on essays from the field with DREAM Charter Schools, which have never been online. There, EnlightenAI agreed closely with DREAM's official scores. For what each state's numbers mean, see How to read state scoring results.
Limits
- 94 graded essays is a modest sample. Domain figures carry a margin of roughly ±9 points.
- The anchor sets don't include the reading passages, so EnlightenAI couldn't check whether evidence was accurate. Florida's scorers can.
- Older papers are paired with the current rubric, which may differ slightly from the version used to score them.
- Anchor papers are chosen by Florida's committees to show each score point clearly; Florida's human and engine figures come from everyday field-test essays.
Sources:
FAQs
Quadratic weighted kappa (QWK) is the standard way state testing programs measure how closely two graders agree. It runs from 0 to 1. It gives more credit for close misses than far ones, and a grader that gives every essay the same score gets 0, however often it happens to match. Above 0.6 is considered good and above 0.8 is excellent.
Two trained scorers reading the same essay don't always agree. States publish how often they do, and that is the fairest bar for any grader: if EnlightenAI agrees with the official score as often as two of the state's own scorers agree with each other, it is grading to the state's standard.
Yes. Each study has a verification kit with every graded response, the examples used, our scores and feedback, and a script that recomputes the tables. Get in touch to request one.
