Robots.txt builder and tester
Test whether a supplied robots.txt permits a crawler to visit a URL.
Bookmark this page
Keep this page handy for next time.
- On a computer
- Press Ctrl + D, or Command + D on a Mac.
- On a phone or tablet
- Open your browser's menu or Share menu and look for Add Bookmark, Bookmark or Add to favourites.
A bookmark saves the page link. Save any tool results separately.
How to use this tool
- Test whether a supplied robots.txt permits a crawler to visit a URL.
- Adjust the labelled settings, then run the tool. Copy the result or download it as a file.
Things to know
- Processing is local. Supplied code and network requests are never executed. Input and text settings are limited to 5 MiB combined; text settings alone are limited to 1 MiB.
- Rules: 512 KiB, 10,000 lines, 2,048 characters per line. Test URL: 8 KiB; crawler: 256 bytes. A three-second worker deadline bounds expensive wildcard matching.
You might also need
More in web developmentCSS selector tester
Find matching elements in pasted HTML without rendering the supplied page.
Regex tester
Find matches, inspect capture groups and test replacements using JavaScript regular expressions.
Semantic version range tester
Check which semantic versions satisfy a package version range.
URL parser and builder
Inspect a URL and edit its path, query parameters or fragment.
Sitemap builder and validator
Create a sitemap from URL lines or inspect a pasted XML sitemap.
Content Security Policy builder
Build a policy from explicit directives and check common unsafe settings.
Cron builder and explainer
Preview the next occurrences of a five-field cron expression.
Gitignore builder
Combine locally bundled ignore rules for a project.
Meta-tag builder and preview
Generate escaped page and social metadata from your title and description.