Abstract
Evaluating components for non-deterministic artificial intelligence agents can be challenging. A computer-implemented system, such as a server or distributed computing system, can use a multi-stage pipeline to programmatically validate machine-generated evaluation scenarios prior to use. For example, the process can involve an oracle gate to test for basic solvability and a dispersion-gated calibration stage. This calibration stage may repeatedly execute a control agent against a scenario and can analyze the resulting score variance, rejecting scenarios with low or flat variance that may be non-discriminating measuring instruments for a stochastic agent. This methodology can produce a validated set of test scenarios, which may be used in formal studies to aid in generating a quantitative assessment of a component's impact on agent performance.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Shirey, Joe and Weinmeister, Karl, "A Pipeline for Calibrating AI Evaluation Scenarios Using Dispersion Analysis", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/12560