Abstract
While sensor-driven systems such as motion detection, home security systems, etc. are capable of preconfigured tasks, such systems offer limited customization. Large language models are capable of performing reasoning tasks based on data provided to them. This disclosure describes a comprehensive semantic sensing system that is configured to understand user intent and to perform reasoning tasks based on different streams of sensor data (e.g., obtained from low-level sensors) to fulfill that intent. Reasoning tasks can be performed using a foundational AI model. Semantic sensors - sensor-systems that utilize a multimodal language model that interprets sensor data - are described that enable systems that can be easily customized, debugged, and used in various applications. Per the techniques, data from sensors (together with user-permitted contextual data) are provided to a multimodal language model that performs reasoning to generate a high-level understanding of the environment in which the sensors are deployed. The understanding can be used to perform custom tasks, answer natural language questions, to debug/configure sensors, etc. Applications of the described semantic sensors include smart homes, environmental monitoring, industrial process control, healthcare, etc.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Olwal, Alex; Liu, Michael Xieyang; Petridis, Savvas; Tsai, Vivian; Fiannaca, Alexander; Terry, Michael; and Cai, Carrie, "Semantic Sensors: Multimodal Language Model Powered Sensors Capable of Reasoning", Technical Disclosure Commons, (June 08, 2025)
https://www.tdcommons.org/dpubs_series/8210