Sara-AnnSansone135

Sometimes, we possibly may want search engines to not index certain parts of the site, as well as exclude other SE from the site all together.

The place where a simple, small 2 line text file called robots.txt is available in this really is.

Once we've a website up and running, we need certainly to be sure that all visiting search-engines can access all the pages we want them to look at.

Sometimes, we may want se's to not list certain parts of the site, or even prohibit other SE from the site all together.

This really is where a simple, little 2 line text file called robots.txt is available in.

Robots.txt rests in your web sites primary directory (on LINUX systems this really is your /public_html/ directory), and looks something like the following:

User-agent: *

Disallow:

The very first line controls the bot that'll be visiting your site, the next line controls if they're allowed in, or which elements of the site they're not allowed to visit

If you like to deal with multiple robots, then the above lines are repeated by simple.

Therefore an example:

User-agent: googlebot

Disallow:

User-agent: askjeeves

Disallow: /

This may enable Goggle (user-agent name GoogleBot) to go to every service and page, while at the same time banning Ask Jeeves from the website fully.

To locate a fairly up to date set of robot user names this visit if you like to permit every robot to index every page of your site, its still very advisable to place a robots.txt file on your site. It will stop your problem records replenishing with entries from search engines attempting to access your robots.txt file that doesnt exist.

The total set of sources about robots.txt at, to learn more on robots.txt see discount oc pool service