Agent Extraction API: robots.txt crawl permission check

Check whether a URL may be crawled per the site's robots.txt for a given user-agent. Returns allowed, matched rule, crawl-delay, sitemaps.

1 route settlement-verified · $0.003 USDC/request · 1 endpoint · Tools · base network · x402 protocol

Verified routes

• GET /api/robots-check · $0.003 · 2026-10-26 · receipt

Other routes are unverified.

Networks: Base (settlement-verified)

Discovery is free. Agents search the catalogue here. Payment happens on the listing 402, not on PayAPI.

url https://payapi.market/mcp

Listing facts

Provider

FreeTier Agent API

Hosted at

freetier-agent-api.pages.dev

Category

Tools

Price

$0.003 USDC per request

Payment networks

Base

Settlement-verified

1 route, last checked 2026-10-26

Listed on PayAPI Market

2026-10-26

Try Agent Extraction API: robots.txt crawl permission check

Call a verified route without paying first. The server should answer HTTP 402 Payment Required with the price, the USDC asset, the network and the wallet to pay. An x402 client pays that amount on Base and retries with the payment header.

curl -i "https://freetier-agent-api.pages.dev/api/robots-check?url=https%3A%2F%2Fexample.com%2Fsome%2Fpage&ua=MyAgent&nofree=1"

About this API

Agent Extraction API: robots.txt crawl permission check is listed in the Tools category with pricing at $0.003 USDC/request. One route has been settlement-verified by PayAPI Market on 2026-10-26. View receipt on Basescan.

Payments settle on base in USDC via the x402 protocol. AI agents discover this API through the PayAPI Market MCP server and pay per request without API keys or accounts.

This API exposes 1 endpoint.

Questions about this API

What does Agent Extraction API: robots.txt crawl permission check do?

Check whether a URL may be crawled per the site's robots.txt for a given user-agent. Returns allowed, matched rule, crawl-delay, sitemaps. It is provided by FreeTier Agent API.

How much does Agent Extraction API: robots.txt crawl permission check cost?

Agent Extraction API: robots.txt crawl permission check is priced at $0.003 USDC per request, paid per call with no subscription. PayAPI Market paid 0.003 USDC for one call to /api/robots-check when it verified the listing. The exact amount for each call is in the HTTP 402 response before any money moves.

Do I need an API key or account to use Agent Extraction API: robots.txt crawl permission check?

No. Agent Extraction API: robots.txt crawl permission check uses the x402 protocol, so an agent pays per request in USDC on Base and gets the response, with no API key, account or signup.

Has Agent Extraction API: robots.txt crawl permission check been verified?

Yes, 1 route. PayAPI Market paid for a real call from its own wallet, received real data back and recorded the USDC settlement on-chain, most recently on 2026-10-26.

How does an AI agent find Agent Extraction API: robots.txt crawl permission check?

Through the free PayAPI Market MCP server at https://payapi.market/mcp, using search_apis or get_api, or over plain HTTP at https://payapi.market/agent/search. Discovery is free; the agent pays the provider directly on the listing's own 402.

More Tools APIs · Browse the marketplace · List your own API

Related APIs

• x402-seller · 1 route verified · Pay-per-call x402 utilities on Base USDC: GET /timestamp (ISO UTC clock) and POST /echo. $0.01 per request. Durable host https://x402seller.mooo.com (urllib-clean 402). Backup https://pay.x402seller.ddnsfree.com.
• Money Reactor DeFi Yield Scanner (stablecoins) · 1 route verified · Deterministic agent utility on Base. Always-deliver guarantee, real-time response, settled in USDC via x402. Pay one call, get the answer inline.
• Public HTTPS Certificate Preflight · 1 route verified · Read-only public TLS certificate preflight: expiry, not-before, hostname match and CA authorization from a pinned public IPv4 TLS handshake. No HTTP crawling, private IPs, secrets or ownership certification.
• Public Email DNS Preflight · 1 route verified · Read-only public DNS preflight for a domain: MX, SPF and DMARC observations for developer and agent workflows. No email contents, credentials or private DNS access. Informational metadata only.
• Money Reactor Instant Store: Agent Onboarding Kit Base (RPC + EIP-3009) · 1 route verified · INSTANT DELIVERY - everything an agent needs to operate on Base: verified public RPCs with getLogs chunking limits, USDC contract and the EIP-712 domain for gasless EIP-3009 transfers, minimal buyer code in Python and JS, agent wallet hygiene rules. BASE_CHAIN.json + README inline in the paid response. Settles in USDC via x402.
• Money Reactor Instant Store: x402 Facilitator Directory (field-verified) · 1 route verified · INSTANT DELIVERY - who settles x402 payments today: 5 facilitators field-verified on Base mainnet (PayAI, UVDA, CDP, x402.org public, Circle Gateway) with endpoints, fees, chains, discovery feeds and the v1/v2 quirks that cost real teams days. FACILITATORS.json + README inline in the paid response. Settles in USDC via x402.

More from FreeTier Agent API

• Agent Extraction API: Currency conversion (ECB reference rates)
• Agent Extraction API: Bulk link checker (up to 20 URLs)
• Agent Extraction API: One-call page audit (status, redirects, SEO tags)
• Agent Extraction API: LLM token + cost calculator across models
• Agent Extraction API: Page metadata (title, Open Graph, canonical, JSON-LD)
• View all from FreeTier Agent API

By FreeTier Agent API