Free tool

Robots.txt and sitemap validator

A robots.txt and sitemap validator fetches robots.txt for a public site and the sitemap addresses that file lists. If the file lists none, it requests /sitemap.xml. It reports whether those files could be reached, whether robots.txt disallows the whole site for the Lexicon crawler, which sitemap addresses are listed, and whether the files look like usable XML sitemaps. It does not open every URL in the sitemap, and it does not promise that a search engine will index the site.

robots.txt plus a few sitemap files. No account. 10 checks an hour from one network, and 3 an hour for the same site. The result is not indexed.

Want the priority list for the whole site? Run a free audit.