Abstract
Traditional graphical user interfaces are characterized by complex navigation paths that require users to traverse multiple menus and elements to complete tasks. The implementation of natural language control for such interfaces typically requires the development of dedicated application programming interface (API) layers for every supported function. These layers are resource-intensive to build and maintain. This disclosure describes a method for enabling natural language control of existing web-based interfaces without new API development. A mapping of user interface journeys is first generated in a codified language that describes specific document object model (DOM) interactions. This mapping is provided to a large language model as part of a system prompt. When a natural language command is received, a corresponding codified interaction sequence is produced. This sequence is captured and programmatically executed by a middleware component. Complex user journeys are thus automated while synchronization is maintained between the application state and the visible interface. The technical effort required to provide addressable interfaces is significantly reduced.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
N/A and N/A, "Addressable User Interface via Natural Language Mapping of Graphical Elements", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/10168