- Vercel's Firewall can now log or block AI crawlers and AI assistants separately, using its AI Bots managed ruleset, on all plans.
- Crawlers gather content for indexing and model training. Assistants load pages when a user asks something. Coverage depends on Vercel's maintained bot list.
- Decide who owns this policy, check your current setting, and use the diff step before publishing any change.
Blocking AI bots is two decisions
"Block the AI bots" sounds like one decision. It is two. One is whether your content may be gathered to feed a model's indexes and training. The other is whether an AI assistant may read your page when a customer asks it a question. Vercel has now split these in its firewall. The change is small. The idea behind it is larger.
What Vercel changed
Vercel's changelog, dated October 29, describes an update to the AI Bots managed ruleset inside its firewall product. Site owners can now pick a separate action, log or block, for each of two bot groups.
Vercel's first group is AI crawlers. These are bots that gather content for indexing and model training. The second group is AI assistants. These load a page because a user took an action or ran a search.
Vercel's own example is a site that blocks the crawlers but lets the assistants retrieve content to answer users' questions. The ruleset is available on all plans.
How the setting works
A managed ruleset is a list of rules that the vendor maintains. You choose what happens to each group. You do not keep the list of bot names yourself. Vercel says it adds new bots to the rules as they appear.
In the dashboard, go to the project's Security area, then Firewall, then Rules. Find the Bot Management section and switch on AI Bots. Open that entry's settings with the gear icon. Pick an action for crawlers and one for assistants, then review and publish.
Teams that prefer the command line can do the same with the Vercel CLI. The command vercel firewall diff shows the changes. The command vercel firewall publish applies them.
The two actions are log and block. Logging first is a sensible way to see which bots reach your site before you shut anything out. That is our suggestion, not Vercel's.
How a firewall can tell the two apart
Vercel's announcement does not say how it sorts bots, so what follows is general background. Managed bot rules commonly look at the user-agent, a label each visitor sends with its request. Many also check whether the request comes from network addresses that the bot's operator has published.
Vercel's two definitions suggest what differs between the groups. A crawler gathers content for indexing and training, typically ahead of any particular question. An assistant request exists because one person asked for something just now. The first is about what you hand over. The second is about whether you are present when someone is asking.
A photocopier in the stockroom and a clerk at the counter both handle your documents. Only one does it because a customer is standing there with a question.
Why the split matters
Many sites already tell crawlers what they may visit through robots.txt, a plain text file at the root of a site. We read this feature as a sign that the question has grown beyond that file. It is now a policy question: which kinds of AI use do we accept? Legal, marketing and security teams may each have a stake. A single on/off switch hides that. Two switches force the organisation to say what it wants.
Block both groups and requests from the bots in Vercel's ruleset are blocked. Block only crawlers and, as Vercel's own example puts it, you can allow assistants to retrieve your content to answer users' questions. Coverage depends on Vercel's maintained list. Which setting is right depends on your business, and the source does not say.
What the announcement does not say
It is a short product changelog. It does not list which bots fall into each group. It does not say how Vercel tells them apart, or how complete the list is. It does not say whether assistant visits lead to traffic or citations.
It is also one vendor's feature. One launch is one data point, not a trend. Treat it as a prompt for a policy conversation, not as proof of how the market is moving.
Questions to put to your team
First, who owns this decision? If nobody does, whatever is currently configured, or left unconfigured, is making it for you.
Second, do we host on Vercel, and what is the AI Bots ruleset set to today?
Third, do we want assistants to retrieve our pricing, documentation and support pages when customers ask? Do we want crawlers to collect the same pages for training and indexing?
Fourth, can we log both groups for a period before blocking either? Use the diff step so a second person sees the change before it is published.
Deciding who may answer questions with your content is not the same as deciding who may copy it. Make each choice on purpose.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Vercel.
