Abstract
Systems and methods are disclosed for real-time media localization that may address potential scalability and fidelity limitations of some conventional approaches. A technical solution may involve a multimodal generative pipeline, which can be implemented on a hybrid edge-cloud architecture. For example, a system can separate dialogue from background audio on a client device (e.g., a smartphone, smart television, personal computer, wearable device, or augmented reality glasses). The isolated dialogue may then be processed to transcribe speech to text and to extract paralinguistic features, such as vocal timbre and emotional tone. This information may be processed by a large language model for a context-aware translation. A generative text-to-speech model can then synthesize a new vocal track in a target language that may approximate the original actor's voice and emotional delivery. This process may be used to generate scalable, high-fidelity localized audio content that can preserve aspects of the original performance.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Daftari, Dev and Rajotiya, Harsh, "A Generative System for Real-Time Audio Dubbing with Voice and Emotion Transfer", Technical Disclosure Commons, (August 18, 2026)
https://www.tdcommons.org/dpubs_series/11410