Add the text, code or data.
01SEO & Marketing toolFormatted output
robots.txt Generator
Build a valid robots.txt with the rules you need.
100% FreeRuns instantlyNo watermark
- 1Paste inputAdd the text, code or data.
- 2AdjustChoose formatting or generation options.
- 3RunProcess everything in the browser.
Private by designProcessed on your device where possible.
Works on any deviceDesktop, tablet and mobile browsers.
No installationOpen the page and start working.
Always free coreUse the free plan for everyday tasks.
How to use robots.txt Generator
Choose formatting or generation options.
02Process everything in the browser.
03Copy or download the clean result.
04Your files stay private
Your work is processed locally in your browser where possible and is never added to a Vootkit upload library.
Learn more about privacyrobots.txt is four lines of plain text with the power to remove an entire site from search. Most of the damage done with it comes from one misunderstanding: it controls crawling, not indexing.
What robots.txt Generator does
Generates a robots.txt with the rules and sitemap reference you specify.
Disallow does not remove a page from search results. It stops crawling — and a page that is linked from elsewhere can still be indexed, showing in results with no description because Google was forbidden from reading it. To keep a page out, use a noindex tag and allow the crawl so it can be seen.
What it does and does not do
| <code>Disallow</code> | Requests that crawlers do not fetch |
|---|---|
| <strong>Does NOT</strong> | <strong>Remove a page from the index</strong> |
| To deindex | Use <code>noindex</code>, and ALLOW crawling |
| <code>Disallow: /</code> | Blocks the entire site — one character |
| Sitemap line | Should be an absolute URL |
| Not a security control | The file is public and lists what you hid |
| Case sensitive | Paths are |
| Location | Must be at the domain root |
Detailed steps
- Add the paths you genuinely want left uncrawled.
- Add the absolute URL of your sitemap.
- Check for a stray
Disallow: /before deploying.
Worth knowing
Never use robots.txt to hide sensitive paths. The file is public at yourdomain.com/robots.txt, so a list of admin URLs you did not want found is a directory of exactly where to look — it is the first file an attacker reads. Protect those paths with authentication and leave them out of robots.txt entirely.
Frequently Asked Questions
Will Disallow remove my page from Google?
No, and this is the single most common misunderstanding. It asks crawlers not to fetch the page; if other sites link to it, it can still appear in results — with no description, because Google was not allowed to read it. Use a <code>noindex</code> meta tag and permit crawling so the tag can be seen.
Is it a security measure?
The opposite. The file is publicly readable and effectively publishes the list of paths you would rather nobody visited. Use authentication for anything that matters.
Do all crawlers obey it?
Well-behaved ones do. It is a convention, not an enforcement mechanism, and scrapers ignore it freely. Anything you need to actually block requires server-side controls.
What is the most dangerous mistake?
A stray <code>Disallow: /</code>, which blocks the whole site. It happens most often when a staging file is copied to production, and the traffic loss is total and takes weeks to recover from.
Is robots.txt Generator free?
Yes. The Vootkit free plan includes 5 tool runs a day. Upgrade to Vootkit Pro for unlimited daily use, an ad-free workspace and saved workflows.
Are my files uploaded?
No. robots.txt Generator runs entirely in your browser — your file is processed on your own device and never sent to a server. There is nothing for us to store or delete.
Do I need to install anything?
No. robots.txt Generator works in any modern browser on desktop, tablet or phone. Open the page and start.
How often can I use it? Is there a daily limit?
On the free plan you get 5 tool runs a day. When you reach the limit you'll see a prompt to upgrade, and it resets the next day. Vootkit Pro removes the cap entirely for unlimited daily use.
Recently viewed
This tool processes everything locally in your browser. You can disconnect from the internet after the page loads and it will still work.