Robots.txt role

Discuss about Social Networking

Moderators: sendtogeo, hellow0rld, eugene

Robots.txt role

Postby Psl938 » Fri Jan 18, 2013 11:01 am

This is a file which is uploaded on the root server of the website. It is used to inform the crawlers about the structure of the website. It is also used to restrict the access to specific folder or URL inside the website.
Posts: 12
Joined: Sat Nov 24, 2012 8:24 pm

Re: Robots.txt role

Postby guptaabhijit318 » Tue Feb 05, 2013 8:27 am

Robots.txt file is a special text file that is for eternity located in your Web server's root directory. Robots.txt file contains limits for Web Spiders, telling them where they have authorization to search. A Robots.txt is like significant rules for search engine spiders what to follow and what not to. Robots.txt file is a very important file if you want to have a good ranking on search engines.
Posts: 277
Joined: Thu Dec 20, 2012 11:05 am

Re: Robots.txt role

Postby petersmith5544 » Mon May 20, 2013 12:31 pm

Robots.txt files are one of the ways a SEO company may attempt to prevent certain pages from being indexed.

For example, if someone would like to disallow the pages directory they would insert the following into their Robots.txt file.
User-agent: *

Disallow: /pages/

Posts: 68
Joined: Mon Dec 24, 2012 10:29 am
Location: India

Re: Robots.txt role

Postby Anna Dawson » Sat Jul 06, 2013 4:51 am

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter.
Anna Dawson
Posts: 14
Joined: Wed Jul 03, 2013 5:30 am
Location: Sydney

Re: Robots.txt role

Postby NatashaKnisley » Fri Oct 17, 2014 9:56 am

Robots.txt file helps to tell Search Engine, which pages of your website need to be indexed and which should be avoided.
Posts: 22
Joined: Thu Oct 16, 2014 9:53 am
Location: Mumbai

Re: Robots.txt role

Postby evathomsan » Tue Mar 17, 2015 10:05 am

Robots.txt file or doc allow or disallow search engine to visit it.
Posts: 63
Joined: Sat Mar 14, 2015 6:24 am

Re: Robots.txt role

Postby Sahujain » Mon May 04, 2015 12:41 pm

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to vists a Web site URL, say
Posts: 4
Joined: Mon May 04, 2015 12:31 pm

Re: Robots.txt role

Postby Kasirajan » Thu Jun 11, 2015 2:01 pm

robots.txt will direct the spiders or crawlers which are used by search engeines to crawl and index the websites in search engeine results page....
Posts: 43
Joined: Fri Apr 17, 2015 2:35 pm

Re: Robots.txt role

Postby Poojasu » Fri Jul 17, 2015 9:07 am

Robots.txt play an important role, It allows or disallow search engine to visit website.
Posts: 91
Joined: Wed Jun 03, 2015 10:54 am

Return to Social Networking

Who is online

Users browsing this forum: No registered users and 2 guests