Abstract
Generative video models can produce content that may not consistently adhere to physical laws, which can result in implausible motion such as objects defying gravity. This characteristic may limit their utility in applications that could benefit from physical realism. A physics-guided reinforcement learning framework can be implemented to address this condition. For example, a system may use a physics simulator, such as a digital twin, to generate a physically-consistent trajectory from a text prompt, creating a reference optical flow map. An optical flow map can also be extracted from the video generated by the model. A reward signal, which may be derived from a comparison of the reference and generated flow maps, can be used in a reinforcement learning loop to update the video model. This process can train the model to generate videos with more physically plausible dynamics, potentially improving their suitability for simulations, visualizations, and visual effects.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Weisz, Ágoston, "Digital Twin-Based Reinforcement Learning with Optical Flow for Physically Realistic Video Generation", Technical Disclosure Commons, (April 16, 2026)
https://www.tdcommons.org/dpubs_series/9794