Abstract
Audio-based systems for pedestrian navigation can be limited by reliance on broad geographic proximity rather than a user's specific line of sight, which may result in generic content not aligned with the user’s observations. Systems and methods are described that may address this by fusing sensor data from a hearable device (e.g., containing an inertial measurement unit) and a paired host computing device, such as a smartphone or wearable device, that includes a global positioning system receiver and a magnetometer. This fused data can be used to determine a user's head orientation. The determined heading, along with geographic coordinates, can be used to retrieve corresponding visual information. A multimodal generative model can then process the visual and location data to produce a descriptive narration. This approach can deliver dynamic, hands-free audio descriptions correlated with a user’s visual focus, which may improve a user’s spatial awareness and reduce the need for active screen interaction.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Owens, Irvin; Koopman, Max; Mao, Xiaomi; Huang, Eric; and Sellaiah, Satish, "Orientation-Aware Contextual Audio Generation Using Fused Sensor Data and a Multimodal Model", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/12872