Thread Rating:
  • 0 Vote(s) - 0 Average
Share Thread:
Is robots.txt file enough to block some pages from Search Engines?
#1
Is robots.txt file enough to block some pages from Search Engines? I mean from crawling and indexing.
Reply
#2
No its not enough. Its best that you block them through .htaccess file.
Reply
[-] The following 2 users Like ksmith29's post:
  • danni, lammao
#3
(10-24-2019, 05:52 AM)lammao Wrote: Is robots.txt file enough to block some pages from Search Engines? I mean from crawling and indexing.

Robots.text file only tells search engines or suggest some might say that it can or cannot crawl & index any page. It doesn't necessarily blocks them. .htaccess file is the best option for you as @ksmith29 suggested.
Reply
[-] The following 1 user Likes danni's post:
  • lammao
#4
Can anyone provide me the code for htaccess file?
Reply
#5
Follow this article. https://help.dreamhost.com/hc/en-us/arti...ccess-file-
Reply
#6
I hope the article helped you.
SmartScrapers is a data scraping services provider company.
Reply
#7
If it is linked to from other websites, a page that is disallowed in robots.txt can still be indexed. While Google won't slither or record the substance hindered by a robots.txt document, we could in any case find and file a prohibited URL in the event that it is connected from different puts on the web.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)
[-]
Ad

[-]
Top Articles
100+ Social Bookmarking Sites List
55+ Image Submission Sites List

[-]
Recent Posts
Best web development and website designing company in Dubai UAE
Unlocking the Power of Digital Marketing: SEO, Social Media, and Content Writing I...webdesigninguae — 06:18 AM
Can AI tools replace search engines in terms of giving good answers?
It will be a bubble actually for the people who will use AI tools as a search engine. And...zeus — 05:45 AM
App Developers Perth
Partner with Perth's leading app developers to ignite your success journey. Our custom mo...ethanmitchell — 05:29 AM
Can AI tools replace search engines in terms of giving good answers?
The most basic thing is that we search in the engines so we can get results in the form o...Ambika_chauhan — 05:19 AM
Drving traffic from Facebook groups
Driving traffic from Facebook groups requires engaging content relevant to the group's in...philip46 — 07:29 PM

[-]
Follow us on Facebook