Abstract
Manual QA testing can be a time-consuming, repetitive, and error-prone task. Certain types of bugs that occur in specific combinations of device model, make, and app version, and in specific device states are difficult to reproduce in a test environment. This disclosure describes techniques that leverage artificial intelligence (AI) to automate and scale up on-device bug detection. Per the techniques, screenshots of an app are periodically captured as the app undergoes automated test execution. The captured screenshots are analyzed by an on-device AI model such as a multimodal large language model (LLM) to detect and report image-rendering issues for the app, including app user interface components and other visual elements. The techniques advantageously increase the available compute resources by leveraging devices where the app-under-test is already executing. The techniques also enable bug detection on actual consumer devices and in arbitrary contexts, thus improving coverage to include bugs that are difficult to reproduce in test environments.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Cheng, Shu-Wei; Wang, Fei; and Cunningham, Sam, "Issue Detection on Mobile Apps via On-device Artificial Intelligence", Technical Disclosure Commons, (September 11, 2024)
https://www.tdcommons.org/dpubs_series/7331