LLaMA-VID: An Image is Worth 2 Tokens -- Efficient Long Video Understanding with LLMs

Fri, 01 May 2026 00:00:00 +0000

LLaMA-VID (Large Language and Video Assistant) is an ECCV 2024 research project that tackles the fundamental bottleneck in video understanding with LLMs: token efficiency. While modern LLMs boast context windows of 128K to 200K tokens, previous multimodal approaches consumed 100 to 500 tokens per video frame, making even a short 5-minute video clip computationally prohibitive. LLaMA-VID’s breakthrough is representing each video frame with just 2 tokens – a compression ratio of 50x to 250x over existing methods.

The key insight is that video frames are highly redundant. Much of a frame’s visual information is shared with the surrounding frames: the background, the setting, the lighting. LLaMA-VID introduces a dual-token representation that separates what is stable across frames (the context token) from what is changing (the motion token). This means you can process a 1-hour video at 1 FPS (3,600 frames) using just 7,200 tokens – comfortably within any modern LLM’s context window.

ECCV 2024 on SoloSoft

LLaMA-VID: An Image is Worth 2 Tokens -- Efficient Long Video Understanding with LLMs