TinyFish provides four web APIs for AI agents: Search, Fetch, Agent, and Browser. Search and Fetch are permanently priced at $0 with no credit card requirement, making them a practical default layer for RAG and document retrieval.
Groundlane is an open-source TypeScript remote MCP server (v0.1.0) giving AI agents web_search, web_fetch, and web_extract through a single stable contract, with auth, provider routing, and resource limits kept at the operator boundary.
Hands-on parameter choices and response structures for web_search (ten adapters, RRF merge, dual-provider default), web_fetch (format/render strategies, finalUrl provenance), and web_extract (CSS selector determinism, no implicit LLM step), with verifiable error boundaries.
A four-dimension comparison (determinism, replaceability, identity boundary, operational cost) between Groundlane's controlled remote MCP contract and traditional local approaches (WebFetch, stealth_fetch, puppeteer, requests), with verifiable scenario recommendations.
Hyperbrowser packages Chrome sessions, proxies, stealth, profiles, and recordings behind managed Playwright and Puppeteer APIs. It fits agents that need to scale real-browser work quickly, while profile credentials, anti-bot compliance, and proxy bandwidth costs remain application responsibilities.
Jina Reader turns a known URL into LLM-friendly Markdown; production use still requires explicit rendering, scope, token-budget, validation, and fallback decisions.
Extraction tools cannot be compared by HTTP 200s. The same 20 URLs must be scored for body text, headings, tables, code, links, metadata, noise, latency, and cost. This article publishes the corpus, adapter contract, and gates, but no winner without a same-version raw run across all four paths.
Bright Data splits web data access into four layers: proxies preserve control, Web Unlocker returns unblocked content, Browser API hosts interactive browsers, and Web Scraper APIs or Datasets deliver structured data.
Scrapy separates crawling into the Engine, Scheduler, Downloader, Spider, Item Pipeline, and middleware; it fits high-volume, rule-driven HTTP crawling where you need control over scheduling, throttling, retries, and storage.
Selenium drives real browsers through standardized WebDriver sessions, making it useful for cross-browser workflows, existing test assets, and remote Grid capacity; it can render JavaScript applications, but it does not guarantee bypassing CAPTCHAs or other anti-automation controls.
Scrapy owns crawl flow and data models, Zyte API handles fetching, browsers, and anti-bot infrastructure, and Scrapy Cloud adds deployment, scheduling, and output; each layer can be adopted independently.
AgentQL replaces brittle CSS and XPath selectors with queries shaped like the data you want: `query_data` returns structured values, while `query_elements` returns interactive Playwright locators. The public Starter plan lists 50 free API calls per month, but its payment, hard-stop, and remote-browser reset rules still need to be verified in Billing.
Apify is not a single crawler. It packages scraping programs as Actors, saves reusable configurations as Tasks, triggers them with Schedules, and delivers results through Datasets. It fits teams that do not want to operate queues, schedulers, and workers, but Actor fees, compute, proxies, storage, and transfer all draw from the same platform budget.
changedetection.io is a web-change signal layer: narrow the monitored content, suppress noise, and notify downstream systems only when a meaningful change occurs. It is neither a search API nor a crawler replacement.
Crawl4AI handles retrieval after a URL is known: use JsonCssExtractionStrategy for stable DOMs, and switch to LLMExtractionStrategy only when extraction needs semantic judgment or must tolerate irregular layouts.
Firecrawl puts single-page scraping, site discovery, whole-site crawling, and JSON extraction behind one API. Cloud removes browser, proxy, and worker operations; self-hosting gives infrastructure control, but not the complete Cloud feature set.
Free access is not one model: recurring allowances, balance top-ups, rate-limited access, one-time credits, and self-hosting have different steady-state costs.
Scrapling puts HTTP, Playwright browsers, CSS/XPath extraction, and a Spider API behind one Python interface. Adaptive selectors save element properties and relocate a target by similarity after a layout change, but the output still needs validation.
Tavily exposes Search, Extract, Map, and Crawl through one web API for agents. The free plan includes 1,000 credits per month; basic, fast, and ultra-fast Search cost 1 credit each, while advanced costs 2.
A web retrieval benchmark must evaluate complete tasks, not HTTP 200s: 30 fixed cases across five failure strata and three live channels, measuring answers, citations, freshness, latency, cost, and unnecessary escalation. This article delivers the harness and gates, but no fabricated ranking while the three live channels remain unconfigured.
An agent should not open a browser for every web task: route first to Search or Fetch, then escalate on explicit signals such as status codes, weak content, JavaScript shells, authentication, or challenge pages, with retry, budget, cache, deduplication, and provenance constraints at every step.
From MarkItDown (175k stars, MIT) to curl_cffi (6k stars), a survey of 34 open-source tools for feeding data to AI. Categorized along five axes: whole-site crawling, AI browser agents, document conversion, smart extraction, and anti-detection infrastructure. The key to selection isn't which tool is best — it's scenario matching.
In 2025-2026, websites need to be readable not just by humans but by AI. From llms.txt and Schema Markup to GEO and RAG ingestion pipelines, this post maps out the complete technical landscape for turning your website into an AI-consumable data source.
Standard Playwright gets blocked by Cloudflare. Both playwright-extra + stealth and nodriver can bypass it. The final step is wrapping the solution into an MCP server so AI agents can use it automatically.