LAVIS: Salesforce's Library for Vision-Language AI

Fri, 01 May 2026 00:00:00 +0000

Vision-language AI – models that understand both images and text – is one of the most rapidly advancing areas of artificial intelligence. Salesforce’s LAVIS (Library for Vision-Language Intelligence) provides a unified framework for training, evaluating, and deploying a wide range of vision-language models including BLIP, BLIP-2, InstructBLIP, and ALBEF.

LAVIS is designed for both researchers and practitioners. Researchers get clean implementations of state-of-the-art models with reproducible benchmarks, while practitioners get a streamlined API for applying these models to real-world tasks like image captioning, visual question answering, and cross-modal retrieval.

Supported Models

Model	Tasks	Year	Parameters
BLIP	Captioning, retrieval, VQA	2022	470M
BLIP-2	Captioning, VQA, retrieval	2023	1.2B
InstructBLIP	Instruction-following VQA	2023	1.2B
ALBEF	Retrieval, grounding	2021	210M
ALPRO	Video-language tasks	2022	250M

Model Architecture

flowchart LR
 A[Image] --> B[Vision Encoder<br/>ViT]
 C[Text] --> D[Text Encoder<br/>BERT]
 B --> E[Cross-Modal Attention]
 D --> E
 E --> F{Fusion Strategy}
 F -->|BLIP| G[Multi-modal Encoder]
 F -->|BLIP-2| H[Q-Former]
 F -->|InstructBLIP| I[Q-Former + LLM]
 G --> J[Output]
 H --> J
 I --> J

Each model in LAVIS uses a different fusion strategy. BLIP uses a standard multi-modal encoder, BLIP-2 introduces the Q-Former (a lightweight transformer that bridges vision and text), and InstructBLIP adds a frozen LLM for instruction-following.

BLIP on SoloSoft

LAVIS: Salesforce's Library for Vision-Language AI

Supported Models

Model Architecture