Abstract
This disclosure describes techniques for using a large language model (LLM) or artificial intelligence (AI) agent to automate software quality assurance (QA) and user interface (UI) testing on a physical wearable device, such as a smartwatch or fitness tracker. The techniques reduce manual testing and verify that an on-screen layout (user interface or visual rendering) matches a designer’s specification within a measured allowance (tolerance or pixel threshold). A computing system receives a human-readable test plan, bug report, product requirements document (PRD), or design specification. The LLM agent interprets the input and sends commands through a debugging interface to perform physical device actions such as taps, swipes, and physical button or rotating crown presses, as a person would. The LLM agent observes device responses through live screenshots and view hierarchy data, records the test on video, and generates a report with visual evidence. The computing system may coordinate multiple LLM agents sharing a wearable device using a file-lock mutex (a software lock that lets only one AI agent use the device at a time) keyed to the device’s serial number to prevent command collisions and race conditions. The computing system may also dynamically change the device’s logical display size and pixel density (screen resolution) at runtime without rebooting to simulate different screen sizes on a single device, reducing the number of test devices or virtual emulators needed. The computing system may then compare screenshots with the design specification and flag visual differences (such as text clipping, baseline drift, or margin shifts) exceeding a specified tolerance, while additional LLM critic agents (sub-agents) review execution logs and visual evidence as independent checkers (adversarial verifiers). Aspects of this disclosure include vision-language model (VLM) agents, hardware-in-the-loop
(HIL) testing, agentic QA, AI-driven test automation, automated software regression testing, autonomous bug reproduction, stale bug triage, developer feedback loops, device-concurrency control, viewport emulation (display-size simulation or dynamic viewport morphing), hybrid structural and visual coordinate grounding, structural screen-signature matching, deterministic action replay, graph-based navigation memory, cross-resolution UI seed maps, circular display geometry adaptation, visual regression testing, continuous integration (CI) presubmit gating, and adversarial multi-agent and device-state verification.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Liu, Di and Yuan, Samuel, "LARGE LANGUAGE MODEL AGENTS FOR TESTING PHYSICAL WEARABLE DEVICES", Technical Disclosure Commons, (November 05, 2026)
https://www.tdcommons.org/dpubs_series/12769