Abstract
Automated large-language-model (LLM) judges increasingly score open-ended learner work — a reading inference, a written argument, an arithmetic explanation — and their scores feed high-stakes gates such as a mastery or promotion decision. A grader of this kind can systematically under-read responses expressed in a non-standard dialect (e.g., African American Vernacular English, "AAVE"), English-as-a-second-language (ESL) phrasing, or an atypical-but-valid reasoning path, even when the underlying competence is identical to a standard-dialect response. Raw disparate-outcome checks (comparing mean scores across groups) cannot distinguish this grader bias from genuine ability differences, and making the grader "aware" of subgroup membership to correct for it introduces its own fairness and legal hazards. This publication discloses a subgroup-blind Differential Item Functioning (DIF) harness that (1) scores every response through the production judge exactly as in live scoring, with the subgroup tag structurally unreachable by the scorer — blindness by construction; (2) computes DIF post-hoc by joining the blind reads with an offline subgroup-tag table; (3) conditions effect sizes on human-label ability anchors (matched competence) via a standardization DIF index, classed A/B/C on the adverse side only; (4) runs a register-varied matched-pair red-team that asserts the focal member of each competence-matched pair is not depressed beyond tolerance; and (5) makes the fairness verdict signable only when the evidence is human-labeled AND the pairs are non-synthetic AND zero gate-feeding flags fire AND the red-team is clean — a synthetic-evidence signability lockout that prevents a fairness dashboard from going green on fabricated data. The novelty lies in applying DIF to the grader (not to items), in judge-blindness by construction, in the matched-pair register red-team, and in the evidence-provenance lockout.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Assuncao, gustavo matthew, "Subgroup-Blind DIF Harness for LLM Judges with Matched-Pair Red-Team and Synthetic-Evidence Signability Lockout", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/10931