Fetch any site's robots.txt, read its rules and sitemaps, and test whether a URL is allowed or blocked for Googlebot and other crawlers.
We fetch example.com/robots.txt through Numarr's own server. Robots.txt controls crawling, not indexing.
A single robots.txt line can hide your whole site from search. Our team audits crawlability end to end.
Book a Free CallRobots.txt is a plain text file at the root of a domain (for example example.com/robots.txt) that tells search engine crawlers which parts of the site they may or may not crawl. It's made up of groups, each targeting one or more user-agents, with Allow and Disallow rules and, often, a link to the site's XML sitemap.
An important gotcha: robots.txt controls crawling, not indexing. Blocking a URL stops crawlers from reading it, but the page can still appear in search results if other pages link to it. To keep a page out of search entirely, use a noindex meta tag instead, and don't block it in robots.txt.
A text file at your domain's root that tells search engine crawlers which parts of the site they can and can't crawl.
No. It controls crawling, not indexing. To keep a page out of search results, use a noindex meta tag rather than a Disallow.
At the root of the domain, at /robots.txt. It only applies to that exact host and protocol.
It blocks crawlers from the entire site for the matching user-agent. Use it carefully, since it can remove your whole site from search over time.
The most specific rule wins, based on the length of the matched path. If an Allow and a Disallow match equally, the Allow takes priority.
Yes. A Sitemap line pointing to your XML sitemap helps crawlers discover all of your pages.
Because robots.txt only stops crawling. A blocked URL can still be indexed from links, which is why noindex is the right tool for removing pages.
Explore the rest of our toolkit or get a hand from our team:
See all free SEO tools →