me added in robots.txt below code:
User-agent: rogerbot
Disallow:
but not fixed and error in my dashboard
Posts made by alixxf
-
RE: Why my site not crawl?posted in Feature Requests
-
RE: Why my site not crawl?posted in Feature Requests
thanks
with below code ? :
User-agent: rogerbot
Disallow:
-
Why my site not crawl?posted in Feature Requests
my error in dashboard:
**Moz was unable to crawl your site on Jul 23, 2020. **Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster
i thing edit robots.txt
how fix that?