Key takeaways
- EnlightenAI matched TEA's official score on 94% of writing short answers. On 2023 test responses, two of TEA's trained scorers agreed with each other 87–94% of the time.
- On reading short answers it matched 86% of the time; TEA's two-scorer range is 77–85%.
- On essays, it landed inside TEA's range for Organization and Development of Ideas, and within one point of the official score on 98.8% of essays.
- The data: 1,013 officially scored responses from every STAAR RLA scoring guide from 2022 to 2026, in English and Spanish.
What we tested
TEA publishes a scoring guide for each released STAAR reading language arts test, with student responses at every score point, the official score and TEA's explanation of it. We collected every guide from 2022 through 2026: grades 3–8 and English I and II, in English and Spanish.
That is 1,013 officially scored responses across 113 prompts. After setting aside five examples per prompt, 452 responses from 101 prompts were graded.
How EnlightenAI compares with TEA's scorers
TEA reports how often two of its trained scorers gave the same score to the same response on the spring 2023 tests. That's the yardstick we use, with one difference to keep in mind: TEA's figures come from everyday test responses, and ours come from released papers, which are clearer. How to read state scoring results explains why that matters.
On both short-answer types, EnlightenAI gave TEA's official score at least as often as two of TEA's trained scorers agreed with each other.
How often each grader gives TEA's official score
Writing short answer (0–1)
Reading short answer (0–2)
Essay: Organization and Development of Ideas (0–3)
TEA scorers: spring 2023 English-language tests, grades 3–8 and English I and II. Essays were read twice; short answers were read twice for 25% of responses. Their bar shows the middle of the reported range. Source: TEA 2022–2023 Technical Digest, Tables 4.7 and 4.8 (pp. 20–22).
Essays are also scored for Conventions (0–2). Here EnlightenAI matched 66.4% of the time in production, just under TEA's 67–76% range, and 75.4% in ensemble mode, inside it. Nearly every miss is one point.
Full results
| Item and trait | Responses | Exact match | Within one point | Agreement score (QWK) | Two TEA scorers |
|---|---|---|---|---|---|
| Writing short answer (0–1) | 83 | 94.0% | 100% | 0.86 | 87–94% · 0.75–0.89 |
| Reading short answer (0–2) | 125 | 86.4% | 100% | 0.87 | 77–85% · 0.76–0.88 |
| Essay: Ideas (0–3) | 244 | 70.5% | 98.8% | 0.83 | 63–77% · 0.74–0.83 |
| Essay: Conventions (0–2) | 244 | 66.4% (75.4% ensemble) | 98.8% | 0.65 (0.75 ensemble) | 67–76% · 0.67–0.77 |
| All items | 452 | 73.5% | 97.6% | 0.78 | – |
Graded a second time with the same setup, ensemble mode returned identical trait scores on 94% of essays.
How we ran the study
We ran every study the way a teacher uses EnlightenAI. For each prompt, five officially scored papers that span the score range act as the teacher's graded examples, with the state's written explanation as their feedback. EnlightenAI then grades every other paper for that prompt once.
- Examples always come from the same prompt. We never mix prompts, grade bands or languages.
- Papers used as examples are left out of the results, and no paper is counted twice.
- Prompts, passages and rubrics are used in full.
- Any calibration note was written on half of the prompts and checked on the other half.
The grader runs in EnlightenAI's production setup at production settings, with no fine-tuning and no second review pass. It writes rubric-based feedback first and then scores. We also report an optional ensemble mode, which writes nine drafts in one request and keeps the middle score on each trait, so regrades come back steadier.
A generous test for any grader
Released state papers are the right place to check that a grader applies the state's standard, but they are a generous test for any grader, including ours.
- They are the clearest examples. State scorers chose them because each one shows a score point cleanly. Everyday writing on a new task is messier.
- They are online. AI models may have seen them in training, which can inflate results.
- They are easy to tune to. Adjusting a grader while looking at the test papers inflates results, so we calibrate on half the prompts and report the other half.
That is why we also test on essays from the field with DREAM Charter Schools, which have never been online. There, EnlightenAI agreed closely with DREAM's official scores. For what each state's numbers mean, see How to read state scoring results.
Limits
- 52 of the graded Spanish responses were transcribed from images in the guides, and some accents may be lost.
- Each prompt contributes only a few graded responses, so the pooled figures are the reliable ones.
- TEA's human figures come from 2023 test responses scored twice; ours come from released papers chosen to show each score point clearly.
Sources:
FAQs
Quadratic weighted kappa (QWK) is the standard way state testing programs measure how closely two graders agree. It runs from 0 to 1. It gives more credit for close misses than far ones, and a grader that gives every essay the same score gets 0, however often it happens to match. Above 0.6 is considered good and above 0.8 is excellent.
Two trained scorers reading the same essay don't always agree. States publish how often they do, and that is the fairest bar for any grader: if EnlightenAI agrees with the official score as often as two of the state's own scorers agree with each other, it is grading to the state's standard.
Yes. Each study has a verification kit with every graded response, the examples used, our scores and feedback, and a script that recomputes the tables. Get in touch to request one.
