Abstract
In fixed-capacity computing environments for artificial intelligence inference, high demand can lead to resource saturation. Conventional methods, such as queuing requests or uniformly degrading computational precision for all workloads, may negatively impact service levels and user experience. A system can dynamically manage computational precision on a per-request basis, where a control plane, such as a gateway, can assess real-time cluster congestion by monitoring signals like memory usage and latency. Concurrently, a classifier can analyze an incoming request's features, such as its task type, to determine its sensitivity to reduced precision. A policy engine can use the congestion level and sensitivity category to select an execution tier. This approach may improve cluster throughput during periods of high demand by shifting tolerant workloads to lower precision, while seeking to preserve a quality of service for sensitive requests.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Grover, Rohan; Kondalam, Satish; Sambi, Muninder; and Ahuja, Aman, "Dynamic Precision Tiering for AI Inference Based on Cluster Congestion and Request Sensitivity", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/12344