04-05-2019, 09:30 AM
So, I leave 5 concepts about these files that must be taken into account and assimilated.
1. Where you place your file is more important than you think.
2. The type and file size can affect that your robots.txt file is not read
3. A disallow only prohibits reading the content not indexing (and first is not focused to desindexar)
4. If the content is not read, as is logical, the HTML directives are ignored
5. The wording of the URLs is simple, but very specific and its reading rules are not as intuitive as it might seem.
1. Where you place your file is more important than you think.
2. The type and file size can affect that your robots.txt file is not read
3. A disallow only prohibits reading the content not indexing (and first is not focused to desindexar)
4. If the content is not read, as is logical, the HTML directives are ignored
5. The wording of the URLs is simple, but very specific and its reading rules are not as intuitive as it might seem.