A robots.txt file gives compliant web crawlers instructions about which URL paths they may request from a website. Our free robots.txt generator helps you prepare those instructions in the correct plain-text format and add the location of your XML sitemap. You can review the generated rules before placing the file on your server.
Robots.txt is useful for crawl management, but it must be configured carefully. A rule that blocks an important folder can prevent search engines from fetching its pages and resources. It is not a security system, and it is not the correct method for keeping confidential information private.
Robots.txt is a publicly accessible text file used by the Robots Exclusion Protocol. It groups crawling rules by user-agent and normally contains directives such as Allow and Disallow. Search-engine crawlers that support the protocol retrieve the file before requesting URLs from the website and apply the rules relevant to their user-agent.
The file must be available at the root of the exact host and protocol it controls. For example:
https://www.example.com/robots.txt
A file located at https://www.example.com/folder/robots.txt does not control the whole website. Rules for a subdomain such as shop.example.com must be served from that subdomain’s own root. HTTP and HTTPS are also separate origins, so the preferred live website should serve its robots.txt file consistently.
You can use robots.txt to ask compliant crawlers not to request selected folders, files or URL patterns. This may help reduce unnecessary crawling of low-value areas such as internal search results, filtered URL combinations or administrative paths. It can also point crawlers to one or more XML sitemaps.
A disallowed URL may still be discovered through links or other sources and can sometimes appear in search results without a descriptive snippet. If a public HTML page must not appear in Google, allow Google to crawl it and use a supported noindex meta tag or HTTP header. If robots.txt blocks the page, the crawler may never see its noindex instruction.
The file is public, its rules are voluntary, and malicious bots may ignore them. Never use it to protect passwords, customer records, private documents or sensitive directories. Use authentication, authorization and appropriate server security. Publishing a sensitive path in robots.txt can even draw attention to its location.
A correct file can support efficient crawling, but creating one does not automatically increase rankings, speed up the website or guarantee indexing. Search visibility also depends on accessible pages, useful original content, consistent canonical signals, internal links and many other factors.
robots.txt.Do not upload a generated file without reviewing it. The generator creates directives from the options you provide; it cannot know which private-looking folders are necessary for rendering or which public pages are important to your business.
User-agent identifies the crawler to which the following group of rules applies. An asterisk represents all crawlers that do not have a more specific matching group:
User-agent: *
You can create separate groups for recognized crawler product tokens when different crawl rules are genuinely required. Keep the configuration simple unless there is a clear reason to separate crawlers.
Disallow specifies a path that the matching crawler should not request. Each path normally uses its own line:
User-agent: * Disallow: /admin/ Disallow: /internal-search/
An empty Disallow: value does not block any path. A value of / blocks crawling of the entire host for that user-agent, so it should never be used accidentally on a public website.
Allow can permit a more specific path inside a broader disallowed area when supported by the crawler. For example:
User-agent: * Disallow: /folder/ Allow: /folder/public-page.html
When rules overlap, matching behavior depends on the crawler’s implementation and the specificity of the path. Test important configurations with the documentation and tools provided by the search engine you care about.
The Sitemap field points to a sitemap or sitemap index. Use the complete absolute URL, including protocol and hostname:
Sitemap: https://www.example.com/sitemap.xml
You may list more than one sitemap when necessary. A sitemap helps crawlers discover preferred URLs, but it does not override a robots.txt block and does not guarantee that every listed page will be indexed.
A hash symbol begins a comment. Comments can explain why a rule exists:
# Block the private administration area User-agent: * Disallow: /admin/
Keep comments concise and avoid publishing sensitive operational information.
User-agent: * Disallow: Sitemap: https://www.example.com/sitemap.xml
User-agent: * Disallow: /admin/ Disallow: /internal-search/ Sitemap: https://www.example.com/sitemap.xml
User-agent: * Disallow: /
A robots.txt block alone is not sufficient protection for a staging environment. The staging site should also require authentication so that unauthorized users and non-compliant crawlers cannot access it.
robots.txt, not robot.txt.Disallow: / can stop compliant crawlers from accessing all URLs.Consider crawl blocking when a section creates many low-value URL combinations, consumes server resources or provides no useful destination for search users. Examples may include internal search results, temporary generated pages, certain filtered navigation paths or private administration routes.
Do not automatically block duplicate public pages as a substitute for proper canonicalization. Search engines need to crawl a page to read its canonical tag. Depending on the situation, a redirect, canonical element, URL cleanup or removal of unnecessary internal links may be more appropriate.
For a page that should remain accessible to users but stay out of search results, use a supported noindex directive and allow crawling until the search engine processes it. For truly confidential content, require authentication instead of relying on robots directives.
https://yourdomain.com/robots.txt in a browser and confirm it returns HTTP 200 as plain text.Disallow rule for accidental matches.If you need to prepare a sitemap, use our XML Sitemap Generator. The sitemap should contain preferred, canonical and indexable URLs rather than every URL the server can produce.
No. A small public website that allows all crawling can work without one. A file becomes useful when you need explicit crawl rules or want to advertise sitemap locations. If the file is missing, crawlers may generally assume that no robots.txt restrictions apply.
It is not a reliable removal method. Blocking the URL prevents crawling, but the URL may still be discovered and indexed from other signals. Use an appropriate noindex, removal process or authentication method according to the situation.
No. It only requests that compliant crawlers avoid the path. Secure administrative areas with authentication, authorization, updates and proper server configuration.
Do not block resources that search engines need to render and understand important public pages. Blocking unnecessary private resources may be acceptable, but test the effect before applying broad folder rules.
No. A sitemap helps discovery and communicates preferred URLs. Search engines still decide which accessible pages to crawl, select as canonical and index.