06-30-2018, 06:04 AM
Major Role of Robots file are :
Before starting to crawl our website, search engine spiders go through the robots.txt file. In this file the bots are execution orders that will allow us:
Deny access to certain pages or directories in order to avoid indexing.
Not allow certain parts of the code of our website to be indexed.
Avoid the indexing of duplicate content caused by parameters or pages for example.
Show the XML sitemap to facilitate the tracking of the web.
Before starting to crawl our website, search engine spiders go through the robots.txt file. In this file the bots are execution orders that will allow us:
Deny access to certain pages or directories in order to avoid indexing.
Not allow certain parts of the code of our website to be indexed.
Avoid the indexing of duplicate content caused by parameters or pages for example.
Show the XML sitemap to facilitate the tracking of the web.