Abstract
Generative artificial intelligence models may face challenges when modifying components within large software systems, as they can lack the project-specific context to link a visual user interface element to its corresponding source code files. A system can utilize a multimodal retrieval-augmented generation approach where historical code revisions, which may include associated user interface images and textual descriptions, are processed with an embedding model to create a searchable vector index. At execution time, a user query, which can comprise a screenshot and a text prompt, may be used to search the index and retrieve semantically similar historical changes. This retrieved information, for example, file paths and code examples from past modifications, can then be used to augment a prompt for a downstream generative model, which may improve the accuracy and relevance of automated code modifications.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Liang, Zheng; Omogbai, Aileme; and Altenhof, Erin, "Multimodal Retrieval-Augmented Generation For Context-Aware Source Code Modification", Technical Disclosure Commons, (January 29, 2026)
https://www.tdcommons.org/dpubs_series/9240