01-12-2018, 01:29 PM
robots.txt is the program that hide important links from the google that is if the website owner dont want to share some information in the website for the public so that they can use the robots.txt by telling that google dont want can crawl that particular link.