12-07-2017, 11:38 AM
Robot.txt file is the important for the websites, so if you have any private are secure pages in your website you are going tell for google through robot.txt file don’t crawl that information it is easy to understand. And you have to hide information from crawlers and robots and bots.