Abstract
A method can be performed during a round of iteration during a loss function optimization process. The method can include a computer obtaining a first matrix and a second matrix. The computer can determine a selected scaling process from a first scaling process and a second scaling process. The computer can modify the first matrix and the second matrix using the selected scaling process to obtain a first modified matrix and a second modified matrix. The computer can determine a first moment and a second moment based on the first modified matrix and the second modified matrix. The computer can determine a gradient using the first moment and the second moment. The computer can update the first modified matrix and the second modified matrix using the gradient. The computer can determine whether or not an optimization threshold has been reached based on the first modified matrix, the second modified matrix, and/or the gradient.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Zhang, Yilang; Yang, Xiaodong; and Cai, Yiwei, "OPTIMALLY SCALED LOW-RANK ADAPTATION FOR EFFICIENT HIGH-RANK FINE-TUNING", Technical Disclosure Commons, (July 01, 2026)
https://www.tdcommons.org/dpubs_series/10782