robots.txt Generator
Generate a valid robots.txt file.
Rules
robots.txt
About the robots.txt Generator
Generate a valid robots.txt file with allow/disallow rules, crawl-delay and sitemap references. Tell search engines what to crawl without hand-writing the syntax.
How to use it
- 1Add the paths you want to allow or disallow.
- 2Optionally set a crawl-delay and your sitemap URL.
- 3Copy the generated robots.txt to your site root.
Guiding crawlers without hand-writing syntax
A robots.txt file tells search engines and other bots which parts of your site they may crawl. It's useful for keeping crawlers out of admin areas, search-result pages, or duplicate paths that would waste your crawl budget, and for pointing them at your sitemap. The syntax is simple but unforgiving — one misplaced slash can block your whole site — so generating it from clear allow and disallow rules avoids costly mistakes. Remember that it's a request, not a lock: it hides nothing, so anything truly sensitive needs real access control.
Common uses
- Blocking crawlers from admin, cart or search-result URLs.
- Pointing search engines to your sitemap for faster discovery.
- Setting a crawl-delay to ease load from aggressive bots.
Frequently asked questions
- Does robots.txt keep a page private?
- No — it only requests that compliant crawlers skip a path. Use authentication for anything that must stay private.
- Where does the robots.txt file go?
- In the root of your site, so it's reachable at yourdomain.com/robots.txt. Crawlers look for it there before crawling the rest of the site.