Abstract
This technology is related to a system and method for multimodal audio generation, editing, and source separation. A computing system receives an audio input, such as an audio mixture, together with conditioning information including text, video or image information, and a reference or target audio sample. The computing system encodes the available inputs into representations, processes the conditioning information, aligns relevant visual information with an audio representation, and uses a generative model to transform an audio latent representation. Based on whether an audio mixture is available and on the conditioning information, the computing system generates new audio or performs mixture-grounded editing, source separation, augmentation, or other audio processing. The computing system decodes the transformed latent representation to generate an output audio signal. The system supports zero-mixture generation and mixture-grounded processing using a common conditional latent model.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Saleem, Muhammad Usama; Tejasvi, Ravi; Xu, Tianyu; Nongpiur, Rajeev; Chatterjee, Ishan; and Kowdle, Adarsh Prakash Murthy, "Multimodal Framework For Music Generation And Audio Editing", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/12261