D. ShinFollow


A vehicle (e.g., automobile, motorcycle, a bus, a recreational vehicle (RV), a semi-trailer truck, a tractor or other type of farm equipment, a train, a plane, a helicopter, etc.) may have displays to make visual presentations of useful information such as maps, personalized routes, media, etc. Often a separate computing device (e.g., a smartphone, a laptop computer, a tablet computer, smartglasses, a portable gaming system, a laptop computer, a smartwatch, etc.) may provide the visual presentation data for the vehicle. Newer model vehicles typically have touchscreen displays, but some vehicles may have displays without an associated touchscreen. In those cases, the user may be required to modify the content on the dashboard through the computing device, which may increase the interaction time and may be cumbersome.

In this disclosure, the computing device may use its camera to determine user interactions with the non-touchscreen display of the vehicle. The camera of the computing device may be pointed toward the non-touchscreen display, and the computing device may apply a perspective correction algorithm with initial user calibration input of dashboard corner zones. A hand tracking model at the computing device may then determine the position of a user’s hand with respect to the non-touchscreen display of the vehicle. A determination of hand contact with the touchscreen may be done using a machine learning model, such as a machine learning calibrated embedding model. The computing device may then compute dashboard screen coordinates for hand contact at the non-touch screen device and update the visual presentation at the non-touchscreen display appropriately.

Creative Commons License

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.