Robots.txt

[jaysh4922]

Robots.txt file is a special text file that is for eternity located in your Web server's root directory. Robots.txt file contains limits for Web Spiders, telling them where they have authorization to search. A Robots.txt is like significant rules for search engine spiders what to follow and what not to. Robots.txt file is a very important file if you want to have a good ranking on search engines.

[RHCalvin]

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

[david0019]

Robots.txt is a simple text file which is used to hide sensitive and private data of the website. For example, every website disallows their admin panel by the robots.txt and by this, they keep their data safe from the hackers.