Abstract
Current techniques to test edge device stacks for LLMs suffer from environment disconnect (for tests executed on the cloud, not on actual device hardware), scalability and throughput limitations, and lack of integrated release gating. This disclosure describes a parallelized evaluation framework for testing large language model (LLM) stacks on edge devices. The framework orchestrates execution of LLM evaluation tasks on a pool of devices via a centralized router. Task datasets are sharded for distributed, asynchronous execution. The framework incorporates a high-performance binary for on-device task execution with programmable inference delays to prevent thermal throttling on the device and a fault recovery system to ensure reliability. The described framework significantly reduces testing time and enables integrated release gating to prevent model performance regressions caused by firmware, compiler, or runtime updates. Designed as a platform-agnostic solution, the framework provides robust infrastructure for stress-testing device processor stacks and validating end-to-end model performance, effectively identifying rare regressions prior to deployment.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Gupta, Mayank; Ghosal, Soham; Aggarwal, Vaibhav; Kumar, Vivek; and Khurana, Rishubh, "Orchestration Framework for Parallelized On-Device Evaluation of Edge Devices", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/10854