Abstract
This publication describes a framework for seamlessly integrating a user’s physical hands into a Mixed Reality (MR) environment by replacing real-world lighting with virtual illumination in real-time. The technique utilizes 3D skeletal tracking to generate a simplified volumetric model of the hand, which is then used to estimate skin thickness and define material properties (albedo and surface texture). By incorporating 3D bone velocity data, rather than standard 2D pixel tracking, the system stabilizes the decomposition process during rapid movements. The resulting maps allow virtual lighting to be applied realistically, simulating effects like light translucency through thin tissue (e.g., finger webbing) while being blocked by bones, resulting in unparalleled visual immersion.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Singh, Ashish and Öztireli, Ahmet Cengiz, "Real-Time Relighting of Physical Hands in Mixed Reality Using 3D Skeletal Tracking and Skin Thickness Estimation", Technical Disclosure Commons, (September 02, 2026)
https://www.tdcommons.org/dpubs_series/11542