YOUR INPUTS · BROWSER-ONLY RESULTS
Robots.txt rule tester
Test a crawler's path with the robots txt tester and see the matching rule. Review retrieval limits and learn what the result can tell you about crawl access.
Open for testing. Contact delivery and tool results are being checked; please review every output before using it.
Send your contact details first. Then use the tool on this device. No account, saved report or report-recovery link is created.
Read the method and limits- 1Contact
- 2Inputs
- 3Result
How do you test robots.txt rules for Googlebot?
Test a path against robots.txt rules for a named crawler. The result shows the rule used and whether retrieval or interpretation was incomplete. Robots.txt controls crawling behavior; it does not by itself establish that a page is indexed, private or eligible for every search feature.
A PRACTICAL WALKTHROUGH
When to use it and what to prepare
For technical editors checking whether a particular crawler is allowed to request a path under supplied robots.txt rules.
- Copy the current robots.txt contents from the intended website.
- Have the crawler’s product token and exact path, including any query string, ready.
Work through these steps
- Paste the rules and indicate whether retrieval succeeded.
- Enter the crawler token and a path beginning with a slash.
- Read the selected group and matching rule before deciding whether a change is needed.
ILLUSTRATIVE EXAMPLE
Put the method into context
Illustrative example: Disallow: /private/ applies to /private/report/. Test an intended exception separately rather than assuming that a broader Allow rule will override every more-specific restriction.
How to interpret the result
The answer is based on the text you pasted. A retrieval failure stays undetermined. A permitted path means these rules do not block that request; authentication, server behavior and indexing controls are separate.
Get a more useful result
- Test both an intended allowed path and an intended blocked path after an edit.
- Keep a copy of the original file so you can review exactly what changed.
Keep these limits in mind
- The tester does not fetch or publish your robots.txt file.
- Robots.txt is not a password system and does not prove that a URL is absent from search.
How this tool works
Pasted rules only. Named crawler groups, longest matching path rule, wildcard and end-anchor support, and Allow on equal specificity. No network request.
Plan the next step
Read the related Intelligence guideReview the service scopeQuestions about this tool
Can I use this tool now?
Yes. Provide your name, email and business or project name, with permission to contact you about the request. The tool opens when Web3Forms accepts the submission. Calculation inputs and results stay in this browser, without saved history.
Does an allowed robots.txt result mean the page is indexed?
No. It describes crawl permission under the supplied rules. Indexing depends on other signals and must be checked separately.
Will this tester change my website’s rules?
No. It evaluates pasted text. Any edit to the live robots.txt file must be reviewed and published separately.
What is the scope of robots.txt rule tester?
Pasted rules only. Named crawler groups, longest matching path rule, wildcard and end-anchor support, and Allow on equal specificity. No network request.