Abstract
Synthetic data is generated to train an audio-video translation model that aims to achieve synchronized audiovisual outputs. Existing translation systems struggle to maintain natural synchronization between translated speech and corresponding visual elements, resulting in awkward timing and reduced content quality. To address this, audio, video, and onscreen text inputs are processed by dedicated models responsible for translation and adaptation tasks. These inputs are translated and adjusted to maintain consistency in timing, tone, and visual synchronization. Manual corrections are applied through human-in-the-loop (HITL) checkpoints, ensuring quality control and addressing nuanced issues that automated processes might miss. The architecture incorporates feedback loops, enabling iterative improvement in both the quality of synthetic training data and the final translated outputs. Such techniques ensure that the translated audiovisual product retains natural synchronization and high fidelity to the original content. The synthetic data generated through this approach is used to train an end-to-end audio-video translation model capable of accurately translating audiovisual content while maintaining seamless integration between audio and video components.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Klein, Daniel V. and McCartney, Paul, "Synthetic Data Generation for Training an Audio-Video Translation Model", Technical Disclosure Commons, (March 20, 2025)
https://www.tdcommons.org/dpubs_series/7926