Granular Indexing Controls

Cloudflare AI Search has introduced more fine-grained controls over the indexing process, eliminating the need for full rescans when only specific content has changed. Developers can now manage their indexed content with greater precision and efficiency.

Reindex Individual Files

Previously, any update to indexed content required a complete rescan of the entire data source. Now, when you know exactly which file has changed or needs to be retried, you can reindex it directly without triggering a full sync.

• Navigate to Overview > Indexed Items
• Select the sync icon next to any file to reindex it immediately
• Changes take effect without rescanning the entire data source

Selective Sitemap Crawling

For websites with multiple sitemaps, you can now specify which sitemap the crawler should use instead of processing all sitemaps listed in robots.txt. This is particularly useful for sites with separate documentation, blog, and product sitemaps.

• Go to Settings > Parsing options > Specific sitemaps
• Enter the URL of the sitemap you want to crawl (e.g., https://example.com/docs-sitemap.xml)
• Trigger a sync to apply the changes
• The crawler will respect your selection up to the maximum files per index limit

These improvements reduce indexing overhead and give you better control over what content is included in your AI Search index, resulting in faster updates and more relevant search results.