Agent Harbor AI-Crawler Access Check
Which AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) a website's robots.txt allows, with fixes.
1 route settlement-verified · $0.01 USDC/request · 1 endpoint · Data · base network · x402 protocol
Verified routes
GET /ai-crawlers· $0.01 · 2026-10-14 · receipt
Other routes are unverified.
Networks: Base (settlement-verified)
Discovery is free. Agents search the catalogue here. Payment happens on the listing 402, not on PayAPI.
url https://payapi.market/mcpAbout this API
Agent Harbor AI-Crawler Access Check is listed in the Data category with pricing at $0.01 USDC/request. One route has been settlement-verified by PayAPI Market on 2026-10-14. View receipt on Basescan.
Payments settle on base in USDC via the x402 protocol. AI agents discover this API through the PayAPI Market MCP server and pay per request without API keys or accounts.
This API exposes 1 endpoint.
How to call this API
Every protected route returns HTTP 402 with x402 payment requirements. An agent pays the stated USDC amount on Base and retries with the payment header. No API keys, no accounts. Browse the marketplace or list your own API.
Related APIs
- Synthetic Universe Data APIs · 1 of 38 routes verified · Pay-per-call data APIs for AI agents on Base, Polygon, Arbitrum, Avalanche, Solana via x402 — $0.01 USDC/call, no API keys or accounts. Live catalog: brianbooms.com/agents/
- GLIN US HTS Duty Evidence · 1 of 3 routes verified · Official USITC HTS classification and published duty-rate evidence by code or product text. Returns General, Special and Column 2 rates, FTA codes, units and footnotes.
- CSV Data Quality Checker · Check a CSV before import: missing cells, duplicate rows, required columns and declared types (string, integer, number, boolean, date), with row/column positions. Comma, semicolon or tab. Up to 1 MiB, 10,000 rows, 100 columns. Deterministic, body not stored; failed executions are not settled.
- x402 Integrity Check · 1 route verified · Call before trusting or paying a third-party x402 seller: checks an x402 listing's price, network and bazaar presence against our sampled ecosystem snapshot, and probes endpoint reachability - flagging dead or mismatched endpoints and prices so an agent can skip them before paying (JSON).
- CanonicalHash — JSON Canonicalization API · Stable deterministic hash of any JSON payload for AI agents. RFC 8785 canonical bytes + SHA-256/SHA-512 digest. Same input, same hash, any runtime. For idempotency + cache keys, dedup.
- Kronos by ForgeMesh · 2 of 22 routes verified · Kronos crypto market intelligence for autonomous agents: hourly signal scores, calibrated price ranges, risk state, decision journals, and outcome audits. Paid per call in USDC on Base via x402.