AI Crawl Control Reference Documentation Now Available

Cloudflare has released new reference documentation for AI Crawl Control, making it easier for developers to manage and monetize access to their content by AI crawlers.

Key Documentation Resources

The release includes three major documentation components:

• GraphQL API Reference — Developers can now query detailed crawler request data, including top paths, referral traffic, and data transfer metrics. The API supports key filters for detection IDs, user agents, and referrer domains.
• Bot Reference — Complete detection identifiers and user agents for major AI crawlers from OpenAI, Anthropic, Google, Meta, and other companies.
• Worker Templates — Ready-to-deploy Cloudflare Worker templates, including an x402 Payment-Gated Proxy for monetizing crawler access or charging bots while allowing free access to human users.

What Developers Need to Know

This documentation enables developers to:

• Implement fine-grained control over which crawlers can access their content
• Monitor and analyze crawler traffic patterns
• Monetize crawler access with payment-gated proxies
• Differentiate between AI bots and human users with accurate detection

The reference materials are now available in the Cloudflare Developers documentation.