Robots.txt Tester

Our robots txt tester lets you paste your robots.txt file alongside a URL path and a crawler user-agent, then instantly shows whether that bot is allowed or blocked. It runs entirely in your browser with no data sent anywhere, so you can sanity-check a directive before it quietly de-indexes a page that should have been live.

The robots txt tester applies real matching rules: it selects the most specific user-agent group, falls back to the wildcard group when needed, and uses longest-match Allow and Disallow logic with full support for the wildcard star and the end-anchor dollar sign. It then highlights the exact rule that decided the verdict, so you are never left guessing why Googlebot or Bingbot can or cannot reach a URL.

Use it to debug a sudden ranking drop, validate a staging-to-production cutover, or confirm a new disallow line before you push it. If you want to keep watch on how those crawl decisions show up in your actual Google Maps and local rankings, you can start free with ProMapRanker.

FAQ

How does the robots txt tester decide allowed or blocked?

It first finds the user-agent group that best matches the bot you entered, preferring the longest specific name and falling back to the star group if none match. Within that group it compares every Allow and Disallow path against your URL, and the rule with the longest matching pattern wins. A tie favors Allow, which mirrors how Googlebot resolves conflicts.

Does it support wildcards and the dollar sign anchor?

Yes. The star character matches any sequence of characters, and a trailing dollar sign anchors the rule to the end of the URL path. So Disallow: /*.pdf$ blocks PDF files specifically, while Disallow: /private/ blocks everything under that folder. The tester translates these into precise pattern checks before judging the path.

Is my robots.txt content uploaded anywhere?

No. All parsing and matching happen locally in your browser using plain JavaScript. Nothing is sent to a server, so you can safely test private, staging, or unreleased robots.txt files without any privacy concerns.