Web scraping APIs for AI agents

Web scraping APIs fetch a public page, follow a crawl, or return the text an agent can read, and the agent pays for that call rather than running its own browser. This page keeps Tools, Data and Intelligence listings that say scrape, scraper, scraping, crawler, crawling or web scrap. A generic extract or convert tool is not included unless it uses those words. The catalogue lookup is free. Paying happens on the provider endpoint over x402, in USDC on Base, Solana or Algorand, with no API key and no account. Settlement-verified means a paid route with an on-chain receipt, and the examples are those claimed listings. Unclaimed (bazaar) imports stay on the page with their label and are not the examples. If you run a scraper, listing it is free and you keep 100% of each settled call. The API count and the endpoint count are the live size of this rule.

19 APIs and 97 endpoints match this page. 18 are settlement-verified: a paid route with an on-chain receipt.

Settlement-verified examples

Web Scraper & Content Extractor API · $0.002 · settlement-verified · On-chain receipt
4 endpoints for HTML scraping: clean main-content text extraction (boilerplate stripped: nav/footer/sidebar/ads), all links split into internal and external with anchor text, full page metadata (title, OG/Twitter tags, canonical, language, author, dates), and structured extraction (heading hierarchy, lists, tables as 2D arrays, images with alt text). Powered by httpx + BeautifulSoup4 + lxml.
AEO Audit API · $0.05 to $0.25 · settlement-verified · On-chain receipt
Scores how visible a website is to AI answer engines (ChatGPT, Claude, Perplexity, Gemini). GET /audit?url=<site> returns JSON: score 0-100, grade A-F, per-section breakdown (schema markup, meta tags, content, technical signals incl. AI-crawler robots rules, llms.txt), issues found, and prioritized fixes. Static-HTML audit, ~2s. Built and operated by Pico, an AI agent, for Synlig Digital (Norway). Lite route: GET /audit/lite?url=<site> at $0.05 returns score, grade, per-section scores and the top 3 fixes. The $0.25 /audit route stays unverified.
Agent Harbor AI-Crawler Access Check · $0.01 · settlement-verified · On-chain receipt
Which AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) a website's robots.txt allows, with fixes.
Agent Harbor Business Website Data · $0.01 to $0.05 · settlement-verified · On-chain receipt
Pay-per-call business website data for agents: validated scrape, enrichment and tracking/consent audit. USDC on Base.
Agent Harbor Website Tracking and Consent Audit · $0.05 · settlement-verified · On-chain receipt
Home-page audit: analytics tags, consent and pre-consent tracking, schema, AI-crawler access, SEO, as ranked fixes.
FastAF · $0.001 to $0.06 · settlement-verified · On-chain receipt
Fast APIs for AI agents: search, scraping, eBay, Maps and more, pay per request with x402 USDC on Base. One free request to try.
Money Reactor Crypto Fear & Greed Index · $0.001 · settlement-verified · On-chain receipt
Crypto Fear & Greed index (0-100) with market classification and 10-day history in one call. Lets trading agents gate entries on market sentiment without scraping. Payment via x402 in USDC on Base.
Public HTTPS Certificate Preflight · $0.005 · settlement-verified · On-chain receipt
Read-only public TLS certificate preflight: expiry, not-before, hostname match and CA authorization from a pinned public IPv4 TLS handshake. No HTTP crawling, private IPs, secrets or ownership certification.
token-budget-saver · $0.002 · settlement-verified · On-chain receipt
Fast, anti-bot protected web scraper and HTML-to-Markdown converter. Uses residential proxies to bypass Cloudflare/bot protections and returns token-optimized Markdown with token-saving metrics. Returns clean 200 JSON even on target timeouts.
Web Search & Data API — SERP, page reader & scrapers · $0.005 to $0.15 · settlement-verified · On-chain receipt
Pay-per-call web search and web data for AI agents: multi-engine SERP, search-and-read, URL-to-markdown scraping, Google Trends, WHOIS/DNS, enrichment, SEC filings, geocoding and academic search.

Unclaimed (bazaar) imports

These rows were imported and the provider has not claimed them. They are not the examples above.

Audit text for malicious AI-skill patterns · $0.005 · settlement-verified · Unclaimed (bazaar)
Audit text for malicious AI-skill patterns. Requires `content`. Nothing to audit — crawling, or discovering this endpoint cold? POST /audit/registry runs the same scanner over the newest servers in the official MCP registry: no input required, $0.005.
Audits a website for AI agent and AI crawler readiness from caller-supplied… · $0.01 · settlement-verified · Unclaimed (bazaar)
Audits a website for AI agent and AI crawler readiness from caller-supplied inputs: robots. Audits a website for AI agent and AI crawler readiness from caller-supplied inputs: robots.txt, HTML, headers, llms.txt, sitemap lastmod, /.well-known. Names the policy per AI token (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Gemini-User, KagiBot, Kagi-User, Applebot-Extended, Bytespider, CCBot, Amazonbot, meta-externalagent) and reports what it could not measure instead of guessing. Makes no outbound request.
Reddit search scraper — keyword-search all of Reddit, or scope to one… · $0.03 · settlement-verified · Unclaimed (bazaar)
Reddit search scraper — keyword-search all of Reddit, or scope to one subreddit, for AI agents. Reddit search scraper — keyword-search all of Reddit, or scope to one subreddit, for AI agents. Find posts matching any query, sorted by relevance, top, new, hot, or comments, with an optional time window. Returns each post’s title, author, subreddit, score, comment count, permalink, external link + domain, flair, and timestamp. No login, no API key. Track brand mentions, monitor topics, and surface Reddit discussion for agents.
Twitter (X) video & image downloader — pass a public tweet / X post URL, get… · $0.005 · settlement-verified · Unclaimed (bazaar)
Twitter (X) video & image downloader — pass a public tweet / X post URL, get direct CDN media links: . Twitter (X) video & image downloader — pass a public tweet / X post URL, get direct CDN media links: .mp4 video URLs with thumbnails plus every image in the post, ready to stream, embed, re-host, or download. No scraping or browser automation.
URL to Markdown API and converter for AI agents — convert any public HTTPS URL… · $0.001 · settlement-verified · Unclaimed (bazaar)
URL to Markdown API and converter for AI agents — convert any public HTTPS URL to clean Markdown, HTML-to-text, and LLM-ready web content with no separate scraper account or API key. URL to Markdown API and converter for AI agents — convert any public HTTPS URL to clean Markdown, HTML-to-text, and LLM-ready web content with no separate scraper account or API key.
Web scraping and extraction credit pack for AI agents: 10 accepted public-URL… · $0.005 · settlement-verified · Unclaimed (bazaar)
Web scraping and extraction credit pack for AI agents: 10 accepted public-URL clean-text reads at a micro-price with no scraper subscription. Web scraping and extraction credit pack for AI agents: 10 accepted public-URL clean-text reads at a micro-price with no scraper subscription.
Web scraping API for AI agents — extract clean text and metadata from one safe… · $0.005 · settlement-verified · Unclaimed (bazaar)
Web scraping API for AI agents — extract clean text and metadata from one safe public HTTPS URL in a single x402 call. Web scraping API for AI agents — extract clean text and metadata from one safe public HTTPS URL in a single x402 call.
Web search that returns structured SERP results as JSON · $0.003 · settlement-verified · Unclaimed (bazaar)
Web search that returns structured SERP results as JSON. Runs on a dedicated SearXNG instance aggregating multiple engines, so there is no per-engine API key, quota or HTML scraping to manage. Each result carries title, URL, snippet, the source engine and a published date where available. Choose general web, news, or image results, up to 10 per call. Use it to ground an answer in current web sources, research a topic, or monitor news on a query.

Questions

Which words put a listing on the web scraping page?

scrape, scraper, scraping, crawler, crawling and web scrap, matched as whole words in the name, description or tags. The listing also has to be in Tools, Data or Intelligence. Exact tags scrape, scraper, crawler and scraping match as well.

How is a scrape paid for?

The agent calls the provider. The provider answers with HTTP 402. The agent pays in USDC on Base, Solana or Algorand over x402 and retries. PayAPI Market is not in that payment.

What is the receipt on a scraping example?

The on-chain transaction for a route PayAPI Market paid. That is what settlement-verified means: a paid route with an on-chain receipt. It covers that route, not every route the listing offers.