Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml
-
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hello There I have a site the problem with this site is I have taken countless backlinks but it is not reflecting on Moz. Is there any way to help Moz find my links?
Hello There I have a site health related..the problem with this site is I have taken countless backlinks but it is not reflecting on Moz. Is there any way to help Moz find my links? herpescurecare.com
Link Explorer | | thelancethealth0 -
How to tell moz about the domain redirects?
If I redirected my previous domain to my new domain, how can I tell moz that the previous domain is redirected?
Link Explorer | | LannaRosse0 -
Can I change the Link Explorer from check with Google US?
I'm trying to check the ranking keywords for a specific article with the Link Explorer, but when I hover over Ranking Keywords on the Overview it tells me that it's looking on Google US. We're a UK based company and the keywords we're trying to rank for are UK based - is there a way of changing this to match?
Link Explorer | | Kestra0 -
Site has DA of 26 but no backlinks
I'm asking this for a guy over at SEOchat: http://forums.seochat.com/new-user-seo-questions-answers-123/unusual-change-domain-authority-486962.html Here's what his post says: Exclamation Unusual Change in Domain authority
Link Explorer | | ThomasHarvey
Guys and Girls, I'm having a weird issue in one of my sites , thing is the site had DA 11 in July and however when check on September Site is having 26 without me doing any type of link building and there are no new posts for this month as well. When checked using open site explorer there are no backlinks indexed in their Mozcape index as well. Site: http://www.beijingtravelchannel.com/ Can you give me any plausible reasons for this to happen, so that i can check them out. Thanks in Advance.0 -
Inbound links that does not appear in open site explorer
There are several links to my website that are old, and are not indexed to locate in Open Site Explorer. Which may be the problem from appearing? This is shown in webmaster tools
Link Explorer | | hectordlux0 -
GTLDs in Open Site Explorer
Curious to know if anyone has run into a similar problem - I entered a site containing a gTLD (.dog extensions) in OSE and it gave me an error (screenshot attached). I checked another tool (ahrefs.com) and was able to extract some data. Does anyone know if these type of domains are not included in OSE and/or why there was an error? Appreciate any insight! RSA04bYsL
Link Explorer | | ATShock0 -
The difference between the statistics MOZ & Ahrefs
Hello dear. I have a website for a year. I made 10K links in one year(from 45 Root Domains).the Ahrefs.com has show full list of link root domains and pages. the ahrefs rank is 49. but MOZ said my Established Links are 5! So I have some questions, please help me: 1- what is difference between MOZ rank and ahrefs rank? 2-When Just-Discovered Link Calculate for my site? 3-why my moz trust is 0.00!?How to improve it? URL: https://esta-register.org
Link Explorer | | allyunit0