Abstract
Map applications in extended reality (XR map-apps) immersively display map data and features within extended reality (XR) environments. Currently, users navigate within an XR map-app using hand tracking and/or gaze tracking, which can be inconvenient. This disclosure describes techniques that leverage large language models (LLMs) to enable users to query an XR map-app using voice commands. With user permission, the user’s XR environment, including the user’s real environment (under XR passthrough) and the user’s virtual environment (e.g., immersive screen, audio, etc.) is shared with the LLM. A question from the user can be answered by the LLM in the context of their real and virtual environments. With user permission, an AI agent can take certain actions such as opening XR applications, opening XR map-app modes such as ground-level walkthrough, aerial flyby, virtual teleportation, etc. The user’s interaction with XR map-apps becomes frictionless and more immersive.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Nguyễn, Tony (Tuấn); Wang, Chunpo; Tsujino, Hiro; and Morvan, Stiven, "Navigating Extended Reality Maps Using Large Language Models", Technical Disclosure Commons, (April 30, 2025)
https://www.tdcommons.org/dpubs_series/8062