Trafilatura: Open-Source Web Text Extraction for LLM Datasets and Research
Extracting clean, structured text from web pages is a foundational task for LLM training datasets, research corpora, and content analysis …
Tags
Python
ScrapeGraphAI: LLM-Powered Web Scraping with Graph Logic
Traditional web scraping is fragile. A scraper built around CSS selectors and XPath expressions breaks the moment the target website updates its …
Rerankers: A Lightweight Python Library Unifying Ranking Methods for RAG Pipelines
Building a production-grade Retrieval-Augmented Generation (RAG) pipeline involves many decisions – which embedding model to use, which …
Ray: Universal Framework for Distributed AI and Python Applications
Distributed computing is the hidden tax on AI and data-intensive applications. The logic of your application — the training loop, the batch …
RapidLayout: Open-Source Document Layout Analysis for Chinese and English
Document layout analysis is the critical first step in any document understanding pipeline. Before OCR can extract text, before tables can be …
pypdf: Pure Python PDF Toolkit
When you need to manipulate PDFs in Python without heavy external dependencies, pypdf is the go-to solution. This pure Python library provides …