Search engines are friendly with a file called robots.txt, whenever bots visit your website they do check it. Robots.txt allows webmaster to restrict access or allow access to bots.

robots txt checker tool

What does robots.txt file do?

• A web robot or web crawler which first visits a website it looks up for the robots.txt files. For example when a search crawler looks up http://www.ePressDNA.com it will look up for http://www.ePressDNA.com/robots.txt.
• This robots.txt actually allows and disallows a search engine to look up certain directories of a domain. For instance if you website has hosted a wiki you do not want to be accessible for search engines to look up, then that directory has to be excluded from looking up by a search engine.

About Robots.txt checker

• The website allows you to enter any URL and check the robots.txt file. This scan is usually looks up errors based on what is disallowed.
• It also shows up warning for any matches what might be misunderstood or ignored by search engine crawlers.

This is not really something that I would recommend webmasters to obsess over as usually most robots.txt of many websites are similar