Abstract
Processing long-form, multi-modal video streams can present computational and memory challenges, as certain sampling methods may process redundant data while potentially missing sparse events. A hierarchical framework can perform a query-agnostic condensation of video data into a semantically dense representation using, for example, content-adaptive keyframe selection and hierarchical aggregation. A query-guided filtering stage may subsequently use a cross-attention mechanism to select visual and other modal tokens relevant to a user's textual query. This approach can reduce the volume of data presented to a downstream reasoning model, such as a large multimodal model. This may improve the allocation of computational resources to semantically salient and query-relevant information for analyzing extended-duration content while managing computational costs.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Tang, Youbau; Wang, Lan; Du, Chen; and Lai, Larry, "Query-Agnostic Condensation and Query-Guided Filtering for Long-Form Video Analysis", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/10975