Abstract
Self-attention mechanisms in neural network models can result in quadratic time and memory complexity, creating a memory bottleneck when processing long sequences. A hierarchical approximation of self-attention can address this limitation. The historical key-value cache can be represented as a balanced hierarchical tree where internal nodes store aggregated representations of token clusters. A query-dependent traversal mechanism can then approximate the influence of distant clusters by pruning subtrees based on a configurable criterion, such as the ratio of a cluster's spatial diameter to its distance from the query. This approach can reduce the computational and memory overhead associated with standard self-attention while operating to preserve contextual information from the historical sequence.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Gulli, Antonio and Florissi, Patricia, "Hierarchical Approximation of Self-Attention Using a Tree-Based Key-Value Cache", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/11768