Direct answer: Cloudflare’s new AI crawler defaults take effect on September 15, 2026. For new domains, Training and Agent bots are blocked by default on pages that display ads, while Search remains allowed. The important SEO catch is that a crawler with both Search and Training classifications can be blocked when Training is blocked. Site owners should review the three policies explicitly instead of assuming that allowing Search always keeps every search crawler accessible.
/robots.txt and watch crawl activity.
What changes on September 15?
Cloudflare now classifies AI-related automated traffic by what it does rather than putting every bot in one “AI” bucket. Its three customer-configurable behaviors are:
| Category | What it does | Practical consideration |
|---|---|---|
| Search | Collects or indexes content so it can answer questions later. | Usually allow if organic discovery and referrals matter. |
| Agent | Acts in real time for a person, including chat-fetch and browser-use agents. | Allow when you want agent-assisted discovery or interaction; restrict when automated access is not useful to you. |
| Training | Collects content to train or fine-tune a model. | Decide based on your content-use policy rather than SEO alone. |
For each behavior, Cloudflare provides three choices: Block on all pages, Block on pages with ads, or Allow (do not block). These controls are available to all Cloudflare customers, including the Free plan, according to the official Cloudflare changelog.
The September 15 default for new domains allows Search but blocks Agent and Training on pages where Cloudflare detects ads. Cloudflare also says mixed-purpose crawlers are evaluated against every applicable behavior. Therefore, a bot classified for both Search and Training can be blocked by a Training block even when the Search policy is set to Allow. Cloudflare specifically names Googlebot, Applebot and BingBot when explaining this consequence in its official announcement.
Recommended setting matrix
There is no universal best configuration. Use the closest starting profile below, then confirm it matches your commercial and content-licensing goals.
| Site goal | Search | Agent | Training | Why |
|---|---|---|---|---|
| Traffic-first blog or publisher | Allow | Allow or block on ad pages | Allow, or block only after checking mixed-purpose impact | Protect discoverability first; make the training decision deliberately. |
| Business or service website | Allow | Allow | Block on all pages or ad pages | Search and user-directed agents may help prospects find the business; model training may offer no direct value. |
| Private documentation or gated research | Block where appropriate | Block where appropriate | Block | Access control matters more than public discovery. Do not rely on crawler controls as authentication. |
| E-commerce catalogue | Allow | Allow if agent shopping is useful | Policy-dependent | Product discovery and agent-assisted shopping can be valuable, while unrestricted data collection may not be. |
Traffic-first recommendation: keep Search allowed. Before blocking Training, inspect which crawlers Cloudflare places in more than one category and verify that the resulting behavior will not remove search access you rely on. Cloudflare’s rule is restrictive: when multiple policies apply, the blocking policy wins.
How to configure Cloudflare AI bot policies
- Sign in to the Cloudflare dashboard.
- Select the account and domain you want to review.
- Open Security Settings.
- Find Configure AI bot policies.
- Review Search, Agent and Training separately.
- For each category, choose Block on all pages, Block on pages with ads, or Allow.
- Save the configuration and record the date, owner and business reason for the choice.
These menu names and mitigation options come from Cloudflare’s current AI bot policy documentation. The legacy Block AI bots setting is marked as deprecating on September 15, so use the three behavior-based controls rather than relying on the old single switch.
The mixed-purpose crawler SEO risk
This is the part most likely to be missed. A setting that sounds like “block model training” may also block a crawler used for conventional search if Cloudflare classifies that same crawler for both activities.
Cloudflare states that mixed-purpose Search-and-Training crawlers are affected by configurations that block Training, including its legacy Block AI bots option. Do not interpret a green “Search: Allow” setting in isolation. Review the Training policy and crawler classifications together.
Safe rollout checklist
- Screenshot the current settings before making a change.
- Change one domain first if you manage several sites.
- Keep Search allowed on a public, traffic-dependent site unless you have a specific reason not to.
- Check mixed-purpose crawlers before applying a site-wide Training block.
- Verify robots.txt still returns HTTP 200 and contains only the directives you intend.
- Monitor crawl activity after the change rather than waiting for rankings or referrals to drop.
- Document the decision so a future administrator does not reverse it without context.
How to verify the change
1. Check robots.txt publicly
Open:
https://yourdomain.com/robots.txtOr run:
curl -i https://yourdomain.com/robots.txtConfirm the response is successful and inspect every User-agent, Allow, Disallow and Content-Signal line. A robots.txt preference and an enforced Cloudflare block are not the same mechanism, so review both.
2. Use AI Crawl Control
In Cloudflare, open AI Crawl Control → Directives. Cloudflare says this view can show robots.txt availability, request patterns, crawler filters and violations. It can also report whether managed robots.txt is enabled and whether Content Signals are present. See the official Directives documentation.
3. Review crawler activity
Use AI Crawl Control to identify which AI services are accessing the site, examine request patterns and set individual crawler policies where needed. Cloudflare lists AI Crawl Control as available on all plans, while Pay Per Crawl remains a private beta feature. The product overview explains the current feature set.
4. Watch your search signals
After changing a traffic-first site, monitor server or Cloudflare logs, Search Console crawl/indexing reports, indexed-page counts and organic landing-page traffic. A same-day check proves the configuration was saved; it does not prove there will be no later discovery impact.
Cloudflare Content Signals example
Cloudflare’s announcement shows a managed robots.txt preference in this form:
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /Cloudflare describes Content Signals as a site owner preference, not a direct block. Enforcement comes from the configured bot policy. Treat the signal as a declaration of intended use and the Security setting as the access-control decision.
Common mistakes to avoid
- Blocking Training without checking mixed-purpose bots. This can affect bots that also perform Search.
- Assuming robots.txt enforces access. Compliant crawlers may follow it, but Cloudflare separately tracks violations and provides blocking tools.
- Using “block everything” as an SEO strategy. A public site cannot gain discovery from crawlers it refuses to serve.
- Leaving the decision undocumented. The setting has business, licensing and traffic implications, not just a technical implication.
- Changing every zone at once. A controlled first-site rollout makes unexpected effects easier to diagnose.
FAQ
What are Cloudflare’s three AI crawler categories?
They are Search, Agent and Training. Search indexes content for later answers, Agent covers real-time user-directed automation, and Training covers collection for model training or fine-tuning.
What changed on September 15, 2026?
Cloudflare set new-domain defaults that block Agent and Training bots on pages with ads while allowing Search. It also applies the most restrictive relevant policy to mixed-purpose crawlers.
Can blocking AI training block Googlebot or BingBot?
Yes, under Cloudflare’s stated classification behavior. Cloudflare says mixed-purpose crawlers such as Googlebot, Applebot and BingBot can be blocked when Training is blocked, even if Search is allowed.
Are these controls available on Cloudflare Free?
Yes. Cloudflare says the behavior-based Search, Agent and Training controls are available to all customers, including Free-plan users.
Where is the setting?
Select the domain in Cloudflare, open Security Settings, and find Configure AI bot policies.
Should a traffic-first website block all AI crawlers?
Usually not by default. Keep Search discovery available, decide separately whether Agent traffic is useful, and assess the mixed-purpose impact before blocking Training.
Bottom line
The September 15 Cloudflare change gives site owners useful control, but the settings are interconnected. For a public site that depends on search traffic, the safest starting point is to allow Search, make an explicit decision for Agent traffic, and verify the mixed-purpose consequences before blocking Training. Save the policy, test robots.txt, and monitor real crawler activity after the change.
Source image credit: Cloudflare. This guide is an independent practical explanation and is not affiliated with or endorsed by Cloudflare.