What is a robots.txt file?

Robots.txt is a plain-text file at the root of your domain that tells web crawlers which parts of your site they may or may not request. It is the first thing most crawlers check before reading your pages.

Should I block AI crawlers like GPTBot?

It depends on your goals. Allowing AI crawlers such as GPTBot, ClaudeBot, Google-Extended, and PerplexityBot lets AI answer engines read and cite your content, which is increasingly how customers discover businesses. Blocking them removes you from those answers.

Do I need a robots.txt file at all?

It is optional. With no robots.txt, crawlers are allowed everywhere by default. A robots.txt is still useful for pointing crawlers to your sitemap and keeping them out of low-value paths like internal search or cart pages.

What does a disallowed path mean?

A disallowed path is a URL pattern your robots.txt asks crawlers to skip. This is normal for admin, cart, or internal pages, but accidentally disallowing your homepage or key sections can stop them from being indexed or cited.

Why is my homepage showing as blocked?

A rule in your robots.txt, often a Disallow line under the wildcard user-agent, is matching your homepage URL. Review the blocking rule and remove or narrow it so search and AI crawlers can reach the page.

Robots.txt Checker

Confirm search and AI crawlers can reach the pages that matter, and see exactly which bots your robots.txt allows or blocks.

This is one slice of the full scan

See your complete website report

This checker covers one part of your site. A full Stackra scan grades the whole thing and shows you the screenshots, expert reviews, and action plan a single check cannot.

Your Growth Readiness Score
One 0 to 100 score across conversion, search, and technical health, with the weak spots ranked.
Expert AI reviews and visuals
A CMO, SEO expert, and CTO read your site, with screenshots and evidence this quick check leaves out.
A prioritized action plan
The fixes that move the needle first, in plain language, tagged by impact and effort.

Free to sign up. Results in about 5 minutes.