Parse the robots.txt file for any website - see crawler rules, blocked paths, and sitemap URLs in a readable format.
Enter a domain to view its robots.txt
robots.txt is a plain-text file that websites publish at their root path (e.g. https://example.com/robots.txt) to tell web crawlers which parts of the site they are allowed or not allowed to index. It follows the Robots Exclusion Standard - a loose convention, not a technical enforcement mechanism. A crawler that respects robots.txt will skip disallowed paths; one that does not (such as a scraper) will ignore them entirely.
Rules are grouped by User-agent - each group applies to a specific crawler (e.g. Googlebot, Bingbot) or to all crawlers using the wildcard *. Within a group, Disallow blocks a path and Allow overrides a disallow for a more specific path. A Crawl-delay directive asks the crawler to wait N seconds between requests to reduce server load. Sitemap lines (which can appear anywhere in the file) tell crawlers where to find the site's XML sitemap.
Understanding a site's robots.txt is useful for SEO audits (are important pages accidentally blocked?), competitive research (which sections does a site want hidden from crawlers?), and debugging indexing issues. A missing or empty robots.txt is valid - it means no restrictions apply.
Sent to our server: Domain, Path, and User agent. We fetch files from that site. To stop abuse, we have to keep a count of your checks for up to 90 minutes; then it is deleted. Also added to the page link, so you can share the result: Domain.