Abstract

A method is disclosed for rating automated code reviewers, such as language-model review agents, from what later happened to the changes they reviewed rather than from the verdicts they gave. Each completed review is joined by change-request number to a build record, the first non-review record in identifier order that links to that change request and yields a branch (not necessarily the one that produced the change), and through that branch to rework records. A review scores a loss when the branch's earliest rework record was created after the review completed, and otherwise a win when the build record is closed; a review whose build is not closed and whose earliest rework does not postdate it, an unlinkable review and a self-review yield no game and are tallied as skipped. Every game is played against one fixed synthetic anchor opponent whose rating and deviation never change, so reviewers who never review the same change share one scale. Each reviewer's games are bucketed into fixed-length rating periods aligned to the run time, empty periods after the first game are retained, and a Glicko-2 update is applied period by period from a default state, so an idle reviewer's deviation grows, while its games remain in the window, as an explicit uncertainty signal. The same games feed a batch Bradley–Terry fit by minorization–maximization, gauged so the anchor has zero log-strength, smoothed by a symmetric pseudo-count prior and reported with a normal-approximation interval. Both models ship dark and fail closed.

Creative Commons License

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.

Share

COinS