Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml
-
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why are my trackable links appearing in my site crawls?
Our Google-created trackable campaign URLs are appearing in site crawls and often triggering error warnings.
Link Explorer | | K.Reeves0 -
My Site Facing indexing issues
Hello, I am facing indexing issues on one of my which is about budget bushcraft knife , Four months have been passed I built my site and published almost 7 article so far. But i am worried no any single keyword ranked on google till yet as i checked through MOZ site explorer. Can anyone guide me what should I need to do now. Thanks
Link Explorer | | james112230 -
Why moz is not crawling my Backlinks & Keyword ?
Hello all brother & sisters, I want to know why moz bot crawler are not detected my backlinks & keyword from last 3-4 months not even single one. All other SEO crawler less popular than moz are detecting my keyword & backlinks. May I know is there any option to forcefully moz bot to crawl my all site specially.
Link Explorer | | loknath.tcms1 -
Why moz and ahrefs reported a remove link that is exist?
Hi in last few days I checked my website in moz link explorer and ahrefs and it showed lots of my backlink as removed! But when I checked this on the browser its OK is there any problem with these tools? Anyone has this problem? Thanks
Link Explorer | | namibiagonzo0 -
Why doesn't Moz crawler follow robots.txt?
It is crawling the entire site, and there is stuff we do not want it to. Please advise.
Link Explorer | | Tylerj0 -
804 error preventing website being crawled
Hi For both subdomains https://us.sagepub.com and https://uk.sagepub.com crawling is being prevented by a 804 error. I can't see any reason why this should be so as all content is served through https. Thanks
Link Explorer | | philmoorse0 -
Weekly and other data not available in moz tools
Dear All, Since last few days we are not getting updated results in moz tools. We get following warnings in the pages. "Hello! Weekly data for some campaigns is a bit behind schedule. Don't worry; it's processing now and will be available soon. Check out http://health.moz.com for the latest status on all of our tools. And thanks for your patience!" How much time it will take to show us the current results, so that we can properly work on it. Right now we are not able to measure any changes affected our site. Thanks, Om
Link Explorer | | omverma0