What changedA site no longer has to treat model-training crawlers and user-triggered AI retrieval as one traffic class inside Vercel Firewall. That enables a more precise policy, but it does not guarantee an assistant will cite or visit the site.
Vercel AI bot firewall rules now have two categories
Vercel announced the change on October 29, 2026. Its AI Bots managed ruleset can now apply separate actions to AI crawlers, which collect content for indexing or training, and AI assistants, which fetch a page in response to a user action or search query.
The Vercel changelog gives a concrete example: block crawlers while allowing assistants. Teams can configure the split in Security, Firewall, Rules, Bot Management, AI Bots, or through the Vercel CLI.
Vercel says the managed rules are updated as new bots are added and the feature is available on all plans. “Managed” reduces signature maintenance. It does not decide which business policy is right for a publisher, SaaS product or private application.
Crawler and assistant access serve different business purposes
A crawler may collect pages at scale for a search index or model workflow. An assistant fetcher may request one page because a user asked a question. Giving both the same rule can block a useful retrieval path while trying to limit large-scale collection.
OpenAI’s official crawler documentation makes the distinction concrete. GPTBot is used for content that may help train foundation models, OAI-SearchBot supports ChatGPT search, and ChatGPT-User performs certain user-triggered actions. OpenAI says the controls are independent.
A documentation site might deny training crawlers, allow or log search and assistant fetches, and protect account routes with authentication regardless of bot identity. The policy follows content purpose, not a simple “AI yes” or “AI no” preference.
| Traffic or route | First action | What to verify |
|---|---|---|
| AI training or bulk crawl | Log, then allow or deny | Volume, routes, robots identity |
| Search and assistant retrieval | Allow or log | 2xx, 403, 429 and referrals |
| Public marketing and docs | Allow with rate controls | Indexing and source attribution |
| Account, admin and private data | Authenticate or deny | No public access path |
Firewall reachability and robots preferences are separate layers
A firewall decides whether a request reaches the application. Robots.txt expresses crawl preferences to bots that choose to follow it. Neither layer alone proves that a page is indexed, surfaced in an answer or cited.
OpenAI says sites can allow OAI-SearchBot while disallowing GPTBot. It also notes that ChatGPT-User may be used for user-triggered actions where robots.txt rules may not apply. That makes route authorization and data exposure essential even when a robots policy is present.
Google documents Google-Extended as a robots control for Gemini training and grounding. Google states that this token does not affect inclusion or ranking in Google Search. The example shows why one global block is too blunt: training preference and search visibility can be independent.
Blocking crawlers does not preserve citations by itself
Allowing an assistant to fetch a page only creates technical reachability. It does not guarantee selection, citation, a referral session or a business outcome.
Imagine a SaaS company that allows assistant retrieval on public documentation but denies training crawlers. The firewall may return 200 to a user-triggered fetch. The page can still be skipped because its answer is unclear, its canonical is wrong, the assistant already has another source or the platform does not expose a link.
Use Vercel AI traffic insights to inspect bot and referral patterns, then connect referral sessions to a website analytics implementation. Keep “accessible to a bot” separate from “selected by an answer system” in every report.
Log before changing the default action
A one-week log window gives teams a baseline for agent names, route patterns, response codes and request volume. Without it, a rule change can solve a hypothetical problem while breaking an existing retrieval path.
Start with public content, account routes and expensive dynamic endpoints as separate groups. A docs page returning 200 to an assistant is expected. A private invoice route appearing in the same log is an authorization problem, not a bot-management preference.
Vercel’s bot management documentation explains the managed rules and its managed rules documentation covers rule actions. Review the diff before publishing a Firewall change, then watch 2xx, 403 and 429 responses after release.
- Inventory public, private and high-cost routes.
- Log crawler and assistant requests separately.
- Check robots.txt for each named crawler policy.
- Change one category or route group at a time.
- Verify response codes, indexing, referrals and conversions independently.
Review these limits before publishing a rule
Managed signatures can identify known bots, but identity is not the same as authorization. Sensitive pages still need real access control, and public pages still need a coherent technical SEO audit.
The new split also does not settle licensing, copyright or training rights. Those are policy and legal questions outside a Firewall toggle. The product change only provides more precise enforcement for requests Vercel can classify.
Finally, category membership can change as vendors update user agents and Vercel updates its rules. Record the reason for each action and review the policy periodically rather than treating the first configuration as permanent.
Audit one week of AI bot traffic before choosing the rule
Seuil can help map bot requests to route sensitivity, robots preferences, search visibility and analytics evidence. The useful deliverable is a short access matrix with an owner and verification plan, not a dramatic site-wide block that nobody measures afterwards.