Abstract
Systems and methods are described for stress-testing automated quality evaluators used in recommendation ranking to detect format-driven score gaming. For an item, the system generates format-perturbed variants that preserve semantic content and content-perturbed variants that preserve formatting, validates the variants using similarity and entailment or divergence constraints, and obtains evaluator scores for the item and variants. Format sensitivity and content sensitivity are computed from score differences, and a gaming score is derived as a ratio of sensitivities with an epsilon term. The gaming score may be used to compute an adjusted ranking score using category-calibrated parameters, to monitor item-, producer-, and marketplace-level gaming trends over time with alert thresholds, and to route high-risk items for human audit. Stress-test outputs may also drive evaluator recalibration using adversarial examples and contrastive pairs to improve robustness, measured via an evaluator robustness index.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Anonymous, "Evaluator Stress Testing for Recommendation Quality Gaming Detection", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/10728