Abstract
Autoregressive decoding in machine learning models with large vocabularies can be limited by memory bandwidth, as the output projection matrix may be read from memory for each generated token. A described method can address this by introducing a multi-step decoding process that first speculatively generates a sequence of draft tokens by scoring only a small, dynamic subset of the full vocabulary, while the full model body is still executed at every step.
Subsequently, a deferred, batched verification phase may validate the drafted tokens by computing full-vocabulary distributions for the entire sequence in a single operation. This approach can amortize the cost of accessing the full vocabulary matrix over multiple token generation steps. The process may reduce the per-token cost associated with memory access and can improve decoding efficiency while producing output that can be identical to standard greedy decoding and, for sampling, distributed according to the model's full output distribution.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Yadav, Karan, "Amortizing Vocabulary Projection Cost with Deferred Batched Verification", Technical Disclosure Commons, (October 21, 2026)
https://www.tdcommons.org/dpubs_series/12364