Allow users to edit a robots.txt for their subdomain
To be able to hide some specific pages on the web portal (/users pages) from web crawlers
2
votes
-
s.larin
commented
Adding a sitemap would also be a nice option.