What do robots.txt and XML sitemaps do?
robots.txt is a small text file at the root of your domain that tells search engine crawlers which parts of the site they may crawl. It's useful for keeping bots out of admin areas, internal search results, carts and other pages that waste crawl time. It doesn't hide pages from Google's index; for that you need a noindex tag.
An XML sitemap is a list of the pages you want search engines to find, optionally with the date each was last updated. It helps Google discover new and deep pages faster, especially on new sites or sites with weak internal linking.
In robots.txt mode, pick your platform for sensible defaults, add any paths to block, optionally block AI training crawlers such as GPTBot, CCBot and Google-Extended, and add your sitemap location. In sitemap mode, the tool crawls your site from the URL you enter (up to 200 pages), skips pages marked noindex or canonicalised elsewhere, and produces a valid sitemap.xml. You can also paste a list of URLs instead.
Using the Robots.txt & XML Sitemap Generator for your business
A wrong robots.txt rule can quietly stop Google from crawling pages that should rank. Our technical SEO audit checks crawling, indexing, sitemaps and redirects across your whole site, and our SEO services then build the content and links that need those pages indexed.
How to use the Robots.txt & XML Sitemap Generator
- Choose robots.txt or XML sitemap. Pick the file you want to create.
- Fill in the options. For robots.txt choose your platform and paths; for a sitemap enter your site URL or paste URLs.
- Click Generate. The file is built instantly (crawling a site takes up to a minute).
- Copy or download. Use the Copy button or download robots.txt / sitemap.xml.
- Upload and submit. Upload the file to your site's root and submit the sitemap in Google Search Console.
Why use this generator
Platform presets
Safe defaults for WordPress and WooCommerce, with a warning for Shopify.
AI crawler control
Block AI training bots with one checkbox.
Crawl-based sitemaps
Finds your pages automatically and skips noindex and canonicalised URLs.
Mistake warnings
Clear warnings if your settings would block the whole site.
Common robots.txt and sitemap mistakes
Blocking CSS and JavaScript: Google needs these files to render pages properly. Don't disallow theme, plugin or asset folders.
Using robots.txt to hide pages: a blocked page can still appear in Google if other sites link to it, just without a description. Use a noindex meta tag (and allow crawling) when you want a page removed from results.
Forgetting staging settings: a site launched with Disallow: / left over from development is invisible to search engines. It's one of the most common reasons a new site gets no traffic.
Sitemap housekeeping: only list URLs that return 200, are indexable and are the canonical version. Google ignores the priority and changefreq fields, so we leave them out, and it uses lastmod only when it's consistently accurate. A single sitemap file can hold up to 50,000 URLs. Most CMS platforms and SEO plugins can keep a sitemap updated automatically, which is the best long-term option.
Frequently asked questions
Where do I upload robots.txt?
To the root of your domain, so it's available at yourdomain.com/robots.txt.
Does robots.txt stop pages appearing in Google?
Not reliably. It stops crawling. Use a noindex tag to keep a page out of search results.
Should I block AI crawlers?
It depends on whether you want your content used to train AI models. Blocking training bots doesn't affect normal Google Search.
How many pages can the sitemap generator crawl?
Up to 200 pages. Larger sites should use their CMS or an SEO plugin to generate sitemaps.
How do I submit my sitemap to Google?
Add it in Google Search Console under Sitemaps, and add a Sitemap: line to robots.txt.