Robots.txt Generator
Control how search engines crawl and index your website.
Global Settings (All Robots)
Specific Search Engines
Restricted Directories
What is a robots.txt file?
The robots.txt file is a simple text file placed in the root directory of your website (e.g., example.com/robots.txt). It uses the Robots Exclusion Protocol to tell web crawlers (like Googlebot) which pages or files they can or can't request from your site.
When to use robots.txt
- To prevent crawling of private or administrative directories (like
/wp-admin/). - To keep search engines from indexing internal search results pages.
- To point search engines directly to your XML sitemap.
- To slow down aggressive bots using the Crawl-delay directive.
Important Limitations
Robots.txt is not a security mechanism. It is a public file that anyone can read. Malicious bots often ignore it entirely. Never use robots.txt to hide sensitive information or passwords. If a page must be hidden, use password protection or the noindex meta tag instead.
Frequently Asked Questions
What is robots.txt used for?
robots.txt tells search engine crawlers which parts of your site they're allowed or not allowed to crawl. It's a crawling directive, not a security mechanism — pages you disallow can still be indexed if linked from elsewhere, and the file itself is publicly readable.
Does disallowing a page in robots.txt remove it from Google's index?
Not necessarily — Disallow only blocks crawling, not indexing. If a page is already indexed or linked externally, Google may still show it in results (often without a description) unless you use a noindex meta tag instead.
Where does robots.txt need to be placed?
It must be placed at the root of your domain (e.g. yoursite.com/robots.txt) — search engines will not check any other location.
Should I block bots I don't recognize?
Legitimate search engines (Googlebot, Bingbot) respect robots.txt, but malicious scrapers often ignore it entirely, so robots.txt shouldn't be relied on as a security control for sensitive content.
⚡ Related Developer Tools
View All 30+ Tools →Schema Markup Generator
Generate valid JSON-LD schema markup for Articles, FAQs, Organizations, and Products.
XML Sitemap Generator
Generate structured XML sitemaps ready for Google Search Console submission.
Meta Tag Generator
Generate optimized HTML meta tags for Google Search, Facebook Open Graph, and Twitter Cards.