01
How crawlers use the file
A crawler normally requests /robots.txt before exploring a host and applies the most relevant user-agent group. Support and interpretation can differ, and malicious clients can ignore the file.
- Public plain text
- Host-specific
- Crawler cooperation
02
Allow, Disallow and Sitemap
User-agent selects a crawler group, Disallow identifies paths not to crawl and Allow can identify exceptions. Sitemap lines point to XML sitemap locations.
Rules apply to paths, not confidential permissions. Review slash placement and case carefully.
03
Full-site blocks and staging settings
Disallow: / asks a compliant crawler not to crawl the whole host. It is sometimes used during staging and accidentally left in place at launch.
A blocked URL can sometimes still appear without a useful snippet when linked elsewhere. Use authentication or network access controls for private systems.
04
Publish and verify safely
Keep a copy of the current file, generate the smallest required change and test important public routes. Review search-console tools and server logs where appropriate.
Practical next steps
Checklist
- 01Confirm the correct host
- 02Review every user-agent group
- 03Check for Disallow: /
- 04Reference the canonical sitemap
- 05Do not expose private paths unnecessarily
- 06Test after deployment
Common questions
Questions and answers
Does robots.txt stop indexing?
Not reliably by itself. Use appropriate indexing directives and access controls for the real requirement.
Can it protect an admin area?
No. Use authentication and authorisation.
Where does it go?
At the root of the relevant public host, such as https://example.com/robots.txt.
Need a considered recommendation?
Discuss the website, not just the symptom.
AHANIX reviews each requirement manually. Technical recommendations may depend on access to the website, hosting or DNS configuration.
Continue learning