Web Scraping on SoloSoft

Douyin TikTok Download API: Open-Source Async Social Media Data Scraping Tool

Fri, 01 May 2026 00:00:00 +0000

Douyin TikTok Download API is an open-source, high-performance asynchronous tool for scraping and downloading content from four major Chinese and international social media platforms: Douyin (抖音), TikTok, Kuaishou (快手), and Bilibili (哔哩哔哩). Created by developer Evil0ctal, the project has earned over 5,100 GitHub stars and serves as a goto solution for researchers, content creators, and developers who need programmatic access to short-form video platform data.

Unlike browser extensions or simple downloader scripts, this project provides a complete REST API backend with a web-based user interface, making it suitable for both automated pipelines and interactive use. It handles the increasingly sophisticated anti-crawling measures employed by these platforms, including the X-Bogus and A_Bogus signature algorithms that Douyin and TikTok use to protect their APIs.

LLM Scraper: Extract Structured Data from Web Pages Using LLMs

Fri, 01 May 2026 00:00:00 +0000

Traditional web scraping relies on brittle CSS selectors and XPath expressions that break the moment a site updates its markup. LLM Scraper takes a fundamentally different approach: it uses large language models to understand page content semantically and extract exactly the data you need as structured JSON.

Built by mishushakov, this open-source tool bridges the gap between unstructured HTML and structured data pipelines. Instead of writing and maintaining selectors, you define a typed schema of what you want to extract, and the LLM handles the rest.

How LLM Scraper Works

LLM Scraper supports multiple LLM providers including OpenAI, Anthropic, and local models via Ollama. You provide a URL or HTML content along with a JSON schema describing the data fields you need, and the tool returns a structured JSON object matching your schema.

ScrapeGraphAI: LLM-Powered Web Scraping with Graph Logic

Fri, 01 May 2026 00:00:00 +0000

Traditional web scraping is fragile. A scraper built around CSS selectors and XPath expressions breaks the moment the target website updates its HTML structure. Maintaining scrapers at scale becomes a constant game of catching up with layout changes, restructuring selectors, and re-testing pipelines. ScrapeGraphAI takes a fundamentally different approach: instead of hard-coding extraction rules, it uses LLMs to understand page content semantically and extract the data you actually want.

The core idea is that an LLM – given a page’s rendered content and a description of what to extract – can identify the relevant information without knowing the page’s CSS structure. This makes ScrapeGraphAI scrapers resilient to layout changes. A website redesign that would break a traditional scraper barely registers: the LLM simply reads the new layout and finds the same information.

Trafilatura: Open-Source Web Text Extraction for LLM Datasets and Research

Fri, 01 May 2026 00:00:00 +0000

Extracting clean, structured text from web pages is a foundational task for LLM training datasets, research corpora, and content analysis pipelines. Trafilatura has emerged as the gold standard for this task – a Python library that consistently achieves the highest F-Score among open-source text extraction tools while remaining lightweight, fast, and easy to integrate.

Developed by Adrien Barbaresi at the Berlin-Brandenburg Academy of Sciences and Humanities, Trafilatura goes beyond simple HTML-to-text conversion. It identifies the main content area of a webpage, strips away navigation, headers, footers, ads, and sidebars, and returns only the meaningful textual content. Its crawling capabilities allow it to recursively follow links within a domain, building comprehensive text corpora from entire websites.