Abstract
This disclosure describes a method of making predictability a measured, certified property of AI systems and robots, so that unpredictable behaviour, surprise attacks and loss of control are detected early and met by automatic restriction. An independent predictor, separate from the governed system and its operator, forecasts the governed system's behaviour, such as its next actions, the resources it will use, the tools it will call and the range of its outputs, before the system acts. Each forecast is signed and fixed in a registry before the outcome is known, so it cannot be altered afterwards, and is then scored automatically against what the system actually did, as recorded by its recorder and certified sensors. The running rate of surprise, meaning behaviour outside the forecast range, is compared with a declared budget: when it rises above the budget, the system steps down to a more restricted level automatically, and its accountable manager is alerted. Declared precursors of loss of control, such as attempts to disable oversight, to copy itself, to acquire resources or permissions beyond its task, to conceal actions, or sudden jumps in capability, act as tripwires that stop or restrict the system at once regardless of the surprise rate. Surprise attacks, by the system or against it, are rehearsed in advance on its certified twin through declared adversarial scenarios, and the system's predicted and actual responses are recorded. Every forecaster, whether a twin, a predictor model or a person, builds a public calibration record from its scored forecasts, and poorly calibrated forecasts carry less weight in decisions that rely on them. The predictor is kept out of the governed system's sight and reach so that the system cannot learn to game its forecasts, hidden probe scenarios and capability checks are run at unannounced times, and surprises that occur together across many systems are treated as a possible coordinated attack.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Wang, Edward D. H., "SI Prediction™: Predictability as a Measured and Certified Property of AI Systems and Robots, in which an Independent Predictor Forecasts Each Governed System's Behaviour before It Acts, Forecasts Are Signed and Fixed in a Registry and Scored against What Actually Happened, a Rising Rate of Surprise Steps the System Down Automatically, Declared Precursors of Loss of Control Act as Tripwires, Surprise Attacks Are Rehearsed in Advance, and Every Forecaster Earns a Public Calibration Record", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/13005