Robots.txt role

[Psl938]

This is a file which is uploaded on the root server of the website. It is used to inform the crawlers about the structure of the website. It is also used to restrict the access to specific folder or URL inside the website.

[guptaabhijit318]

Robots.txt file is a special text file that is for eternity located in your Web server's root directory. Robots.txt file contains limits for Web Spiders, telling them where they have authorization to search. A Robots.txt is like significant rules for search engine spiders what to follow and what not to. Robots.txt file is a very important file if you want to have a good ranking on search engines.

[petersmith5544]

Robots.txt files are one of the ways a SEO company may attempt to prevent certain pages from being indexed.

For example, if someone would like to disallow the pages directory they would insert the following into their Robots.txt file.
User-agent: *

Disallow: /pages/

Sitemap: http://www.example.com/sitemap.xml

[Anna Dawson]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter.

[NatashaKnisley]

Robots.txt file helps to tell Search Engine, which pages of your website need to be indexed and which should be avoided.

[evathomsan]

Robots.txt file or doc allow or disallow search engine to visit it.

[Sahujain]

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html.

[Kasirajan]

robots.txt will direct the spiders or crawlers which are used by search engeines to crawl and index the websites in search engeine results page....

[Poojasu]

Robots.txt play an important role, It allows or disallow search engine to visit website.

[riyajindal]

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. ... robots can ignore your /robots.txt.