AI Web Scraping Tools Landscape: A Selection Guide for 34 Open-Source Projects

🌏 中文版

"Scraping data for AI" has spawned an entire tool ecosystem. From MarkItDown at 175k stars to various niche tools in the low thousands, at least 34 active projects on GitHub address this problem. This article categorizes them along five axes — whole-site crawling, AI browser agents, document conversion, smart extraction, and anti-detection infrastructure — to help you pick the right tool for your scenario instead of building a scraper from scratch.

It's Not Just "Download the Page"

Scraping data for AI involves three independent core problems, and no single tool solves all of them:

1. Format conversion: How to turn unstructured web pages or documents into LLM-digestible formats (Markdown, structured JSON)
2. Interactive crawling: How to handle pages requiring login, JS rendering, or dynamic loading
3. Reliable access: How to get data consistently in an environment of increasingly aggressive anti-bot measures

Who Decides How to Scrape: Five Approaches

The biggest divergence among tools is "who's the decision maker":

The five categories below follow these approaches.

Whole-Site Crawling: Firecrawl Leads, but Watch the License

The top pick is Firecrawl (170k stars, AGPL-3.0) — the most feature-complete option with built-in JS rendering, Markdown output, and sitemap scanning. Its API is designed specifically for LLM input scenarios. The catch is the AGPL license: if you use it to provide a network service, your integration code must also be open-sourced.

If the license is a deal-breaker, Crawl4AI (79k stars, Apache-2.0) is the closest alternative — Python-based, lighter weight, and growing fast.

For million-page scale, use Scrapy (63k stars, BSD-3) — the veteran Python framework with a mature distributed architecture, though you write your own parsers. JS/TS teams should look at Crawlee (25k stars, Apache-2.0) by Apify, with clean APIs and Playwright/Cheerio support. Non-engineers can try Maxun (~17k stars), a no-code interface for marking elements to scrape directly in the browser.

AI Browser Agents: Let AI Operate Like a Human

These tools let AI operate a browser autonomously — clicking, filling forms, scrolling, taking screenshots, all self-directed.

Browser-Use (110k stars, MIT) has the largest community. It's a Python autonomous agent loop where every step relies on LLM reasoning — ideal for "let AI complete tasks on the web" scenarios. Stagehand (24k stars, MIT) has the cleanest API — three primitives (act / extract / observe) cover both interaction and data extraction, built on Playwright in TypeScript, better for stable automation scripts. Skyvern (23k stars, AGPL-3.0) takes a vision-first approach — no DOM parsing, pure screenshot-based decisions, best cross-platform capability but slowest per step.

Browser-MCP (~7k stars) exposes browser operations as MCP tools, making it easy to plug into Claude or LLM agent workflows.

For a deep dive into the pure-vision approach, see our Midscene.js analysis — it takes the extreme stance of "screenshots only, no DOM" and even removed its DOM action mode in v1.0.

Document Conversion: No Scraping, Just Format Translation

These tools don't scrape — they work on files you already have. They're in this post because format conversion is the first of the three core problems in "scraping data for AI", but the selection logic for this layer doesn't live in a scraping context, and the internal differences are bigger than they look.

MarkItDown (175k stars, MIT) and anydoc (746 stars, MIT) are pure conversion: they read structure that already exists in the file and map it to Markdown — no model, no GPU, median latency in milliseconds. MinerU (78k stars), Marker (39k stars), and Docling (65k stars) are parsing tools that run models and do OCR, for scanned documents and complex layouts. They are two to three orders of magnitude slower, and their real selection axis is licensing rather than accuracy.

This layer has its own series on the site and isn't duplicated here: Document Parsing in Practice covers how to pick a layer and the licensing traps of the three parsers.

Lightweight options: Trafilatura (~6k stars) specializes in "extract body text from web pages, filter ads" — stable and reliable for preprocessing. Jina Reader (12k stars, Apache-2.0) requires zero setup — prepend r.jina.ai/ to any URL to get Markdown. Readability (~9k stars) is the engine behind Firefox's Reader Mode, often embedded as a preprocessing step in other tools.

Smart Extraction: Self-Healing Selectors

Rule-driven scrapers break on website redesigns. These tools use AI or adaptive mechanisms for more resilient extraction.

Scrapling (76k stars, BSD-3) uses adaptive selectors — no LLM involved, just smart algorithms that automatically repair broken selectors after site redesigns. Fast and token-free. ScrapeGraphAI (29k stars, MIT) takes a different path: describe what data you want in natural language, and it uses an LLM to build the scraping pipeline automatically — great for one-off extraction tasks. AutoScraper (8k stars, MIT) is even simpler — give it a sample page and the data you want, and it learns the selectors itself.

AgentQL (~1k stars) replaces CSS/XPath with semantic queries, Parsera is a lightweight LLM extraction library, and ferret (~6k stars, Go) offers a declarative extraction language.

Anti-Detection & Infrastructure: The Foundation for Reliable Data Access

curl_cffi (6k stars, MIT) spoofs TLS fingerprints so HTTP requests look like they're from a real browser. CloakBrowser (~29k stars) is a stealth Chromium that can drop-in replace Playwright's browser instance. botasaurus (~6k stars) is a Python anti-detection scraping framework, and SeleniumBase (~13k stars) is Selenium on steroids with built-in stealth mode.

changedetection.io (~33k stars) does something different — it monitors web page changes and notifies you, useful for tracking prices, inventory, or policy updates. scrcpy (148k stars, Apache-2.0) isn't strictly a scraper but an Android screen mirroring tool, useful when you need to extract data from mobile apps. brightdata-mcp (~3k stars) is a commercial-grade MCP server for AI agents to access data through Bright Data's infrastructure.

For more on anti-detection techniques, see our guide to bypassing Cloudflare anti-bot (in Chinese) comparing nodriver / stealth / camoufox. For a practical example of connecting scrapers to MCP, see turning a scraper script into an MCP Server (in Chinese).

Quick Reference Table

Major tools verified via GitHub API (re-queried 2026-08-21), sorted by stars:

An additional 17 tools — including changedetection.io, CloakBrowser, Maxun, SeleniumBase, Readability, Browser-MCP, Trafilatura, ferret, botasaurus, AnyCrawl, Markdowner, CyberScraper-2077, brightdata-mcp, webclaw, Parsera, AgentQL, and Craw4LLM — are documented in the research notes, mostly in the 1k–10k star range.

Bottom Line

Tools in this space distribute along two axes: "rules vs. AI" and "general vs. specialized." The 2024–2025 trend is clear: AI-driven scrapers (ScrapeGraphAI, Browser-Use, Stagehand) and document-to-LLM-format converters (MinerU, Marker, Docling) are growing explosively. But rule-driven veterans (Scrapy, Crawlee) remain irreplaceable at million-page scale.

The key to selection isn't "which is best" — it's scenario matching:

• Markdown output + don't want to deal with JS rendering → Firecrawl (watch AGPL) or Crawl4AI
• Login / complex interaction → Browser-Use or Stagehand
• PDF / Office conversion → MarkItDown (general) or MinerU (academic PDFs)
• Site keeps redesigning, selectors keep breaking → Scrapling
• Blocked by Cloudflare → curl_cffi + Cloudflare bypass guide (in Chinese)

Changelog

• 2026-08-21: Re-verified every star count via the GitHub API and fixed three renamed orgs — Firecrawl mendableai → firecrawl, Docling DS4SD → docling-project, Marker VikParuchuri → datalab-to (the old URLs still 301, but the names were wrong). The biggest month-over-month moves were Firecrawl (155k → 170k) and Scrapling (71k → 76k). Also removed curl-impersonate from the anti-detection section entirely — it last pushed on 2024-07-18, two years dormant — and replaced it with the still-maintained curl_cffi. Also shortened the "Document Conversion" section: it used to discuss pure converters (MarkItDown, anydoc) and OCR-performing parsers (MinerU / Marker / Docling) as one group despite a two-to-three order of magnitude difference. It now carries only the boundary and a handoff — the full selection logic, licensing traps and measurements belong to the Document Parsing in Practice series and are not duplicated here.
• 2026-08-06: Added anydoc to the "Document Conversion" section (Firecrawl's Rust conversion library — 14/14 formats, 4.7ms median, 746 stars, MIT licensed), noting how its license differs from the AGPL-3.0 Firecrawl main project. For the full selection logic on this layer, see the document parsing series. Star counts for the other tools in this post remain as queried on 2026-07-24 and were not re-verified.

References

• Firecrawl (GitHub)
• Crawl4AI (GitHub)
• Browser-Use (GitHub)
• Crawlee (GitHub)
• Scrapy (GitHub)
• MarkItDown (GitHub)
• Scrapling (GitHub)
• ScrapeGraphAI (GitHub)
• Stagehand (GitHub)
• Skyvern (GitHub)
• MinerU (GitHub)
• Marker (GitHub)
• Docling (GitHub)
• anydoc (GitHub)
• Jina Reader (GitHub)
• Trafilatura (GitHub)
• AutoScraper (GitHub)
• curl_cffi (GitHub) — maintained TLS fingerprint spoofing
• changedetection.io (GitHub)
• scrcpy (GitHub)
• Midscene.js: Vision-First UI Automation (on this site, in Chinese)
• Bypassing Cloudflare Anti-Bot Guide (on this site, in Chinese)
• Turning a Scraper into an MCP Server (on this site, in Chinese)