Vision-Language on SoloSoft

LAVIS: Salesforce's Library for Vision-Language AI

Mon, 01 Jan 0001 00:00:00 +0000

Vision-language AI – models that understand both images and text – is one of the most rapidly advancing areas of artificial intelligence. Salesforce’s LAVIS (Library for Vision-Language Intelligence) provides a unified framework for training, evaluating, and deploying a wide range of vision-language models including BLIP, BLIP-2, InstructBLIP, and ALBEF.

LAVIS is designed for both researchers and practitioners. Researchers get clean implementations of state-of-the-art models with reproducible benchmarks, while practitioners get a streamlined API for applying these models to real-world tasks like image captioning, visual question answering, and cross-modal retrieval.

Supported Models

Model	Tasks	Year	Parameters
BLIP	Captioning, retrieval, VQA	2022	470M
BLIP-2	Captioning, VQA, retrieval	2023	1.2B
InstructBLIP	Instruction-following VQA	2023	1.2B
ALBEF	Retrieval, grounding	2021	210M
ALPRO	Video-language tasks	2022	250M

Model Architecture

flowchart LR
 A[Image] --> B[Vision Encoder<br/>ViT]
 C[Text] --> D[Text Encoder<br/>BERT]
 B --> E[Cross-Modal Attention]
 D --> E
 E --> F{Fusion Strategy}
 F -->|BLIP| G[Multi-modal Encoder]
 F -->|BLIP-2| H[Q-Former]
 F -->|InstructBLIP| I[Q-Former + LLM]
 G --> J[Output]
 H --> J
 I --> J

Each model in LAVIS uses a different fusion strategy. BLIP uses a standard multi-modal encoder, BLIP-2 introduces the Q-Former (a lightweight transformer that bridges vision and text), and InstructBLIP adds a frozen LLM for instruction-following.

Seed1.5-VL: ByteDance's Vision-Language Foundation Model Achieving 38 SOTA Benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

In the rapidly advancing field of vision-language models, a new heavyweight has emerged from an unexpected corner. Seed1.5-VL, developed by ByteDance’s Seed team, has achieved state-of-the-art results on an astonishing 38 out of 60 public benchmarks, spanning image understanding, video comprehension, document parsing, and multi-image reasoning.

Built on a 20-billion parameter Mixture-of-Experts (MoE) architecture with approximately 2 billion activated parameters per token, Seed1.5-VL represents a careful balancing act between raw capability and computational efficiency. It outperforms models with far larger parameter counts while maintaining inference speeds suitable for real-world applications.

The model’s benchmark sweep is remarkable not just for the number of wins, but for the breadth of categories it dominates. From OCR and chart understanding to multi-image reasoning and video comprehension, Seed1.5-VL demonstrates that ByteDance’s research team has achieved something genuinely comprehensive in the multimodal space.

SGLang Omni: Multimodal LLM Inference with SGLang

Mon, 01 Jan 0001 00:00:00 +0000

Multimodal AI — models that understand images, audio, and video alongside text — has moved from research novelty to production necessity. Document processing systems need to extract information from PDFs and screenshots. Content moderation platforms need to analyze images and video frames. Accessibility tools need to transcribe and describe audio content. Each use case requires an inference engine that can handle the computational demands of multimodal models.

SGLang Omni extends the SGLang inference framework to support these workloads. It adds vision encoders, audio processors, and multimodal token generation to SGLang’s structured generation and high-performance inference capabilities. The result is a multimodal inference engine that not only runs vision-language and audio models efficiently but also produces structured, constraint-compliant outputs — turning image content into parseable data.