Platform guide
How Claude finds and cites websites
Claude can search the web, and Anthropic says each answer that uses search includes citations so people can check the sources. Anthropic runs three agents: Claude-SearchBot for search quality, Claude-User for pages a user asks Claude to read, and ClaudeBot for training. Allowing Claude-SearchBot is the documented step for being found; how Claude ranks results is not published.
Anthropic’s three agents
| Agent | Purpose | Effect of blocking |
|---|---|---|
| Claude-SearchBot | Indexes content to improve search results | Anthropic says it may reduce your visibility and accuracy in Claude’s search answers |
| Claude-User | Fetches a page when a user asks Claude to | Claude cannot read the page on a user’s request |
| ClaudeBot | Collects content for model training | Training use stops; separate from search |
What independent testing suggests
Independent comparisons have found Claude’s cited results resemble Brave Search results, and Brave appears on Anthropic’s subprocessor list. Anthropic has not confirmed its search index, so treat this as a strong hint, not a fact. Being indexed by Brave, Bing and Google covers the likely routes.
Checklist
- robots.txt allows Claude-SearchBot and Claude-User.
- Pages load without heavy scripts hiding the main text.
- Facts on the page are specific and sourced, so a cited passage stands on its own.
- Your business facts are consistent across the web.
Checked against the company’s own documentation on 16 November 2026. Ranking methods are not published by any of these companies; anything beyond the documented controls is informed estimation, not fact.
Related services
Sources
Primary references this guide relies on. Provider documentation changes, so check the source before acting.
Questions
It searches when a question needs current information and web search is enabled; Anthropic’s help centre explains the setting.
Ready to talk through your project?
Send a voice note, spreadsheet or a plain description of where your business is struggling. We’ll help you find the right next step.

