Robots.txt Tester
Paste robots.txt rules and test whether a URL path is allowed or disallowed for a given user-agent.
Test robots.txt Allow and Disallow rules for a URL path
The robots.txt protocol
tells crawlers which paths they may fetch. Before you deploy changes, paste your file, pick a user-agent token,
and enter a path such as /blog/post-1. The tester reports whether the path is allowed or blocked
and which rule won when several patterns match.
Matching rules used here
User-agent lines use case-insensitive substring matching (so Googlebot matches
Googlebot/2.1). Path rules follow common crawler behavior: the longest matching rule wins, and
Allow beats Disallow when two rules have the same length. Wildcards *
and end anchors $ are supported in paths.
To author a new file, start from the Robots.txt Generator, then validate paths here.
Frequently Asked Questions
Does this fetch my live robots.txt from the web?
No. You paste the rules manually. This avoids CORS limits and keeps the check entirely on your device.
Will results match Google Search Console exactly?
Major crawlers agree on most rules, but vendor-specific quirks exist. Treat this as a fast sanity check before you publish, not a guarantee of every bot's behavior.
What path format should I use?
Use the URL path starting with /, for example /private/report.pdf. Query strings are not part of robots path matching in this tester.
Related Tools
Matrix LU Decomposition Calculator
Post Office Monthly Income Scheme Calculator
Pennies to Dollars Converter
Yes or No Generator
Pi Attenuator Calculator
Earned Value Calculator
SOAPAction Header Builder
GPT Bot Crawl Tester
JSON-LD To Flattened
AES GCM File Encrypt Decrypt
LESS to SCSS Converter
Generate Magic Calendar Dates
McWorter Dendrite Fractal
Certificate of Deposit Calculator
Word Frequency Counter
Day of Year Calendar
Python Formatter
Horizontal Projectile Motion Calculator
Cups to Pints Converter
JSON to INI Converter