Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml
-
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
403 errors in Moz but not in Google Search Console
Hello, Moz is showing that one of the sites I manage has about ten 403 errors on main pages, including the home page. But when I go to Google Search Console, I'm not getting any 403 errors. I don't know too much about this site (I handle the SEO for a few sites as a contractor for a digital marketing agency), but I can see that it's a WordPress site (I'm not sure if that's relevant). Can I assume this a Moz issue only? Thanks, Susannah Noel
Link Explorer | | SusannahK.Noel0 -
Learn how to use Open Site Explorer to find opportunities in your links' anchor text. Get your Daily SEO Fix!
In today's Daily SEO Fix, Tori dives into using Open Site Explorer to find opportunities—particularly potential keyword ranking opportunities—in your links' anchor text. Watch "How to Use Open Site Explorer to Check Your Anchor Text" to get started! The Daily SEO Fix is an ongoing series of Moz tool tips and tricks in under 2 minutes. To watch all of our videos so far, and to subscribe to future ones, make sure to check out the Daily SEO Fix channel on YouTube!
Link Explorer | | MattRoney4 -
Link Acquisition Assistant redirect to Open Site Explorer??????
Link Acquisition Assistant redirect to Open Site Explorer?????? How can i use Link Acquisition Assistant?
Link Explorer | | bondhoward0 -
Is there a preferred Browser for Moz?
I having difficulties using Firefox on the Mac. Crawl Diagnostics results never show up. I see the numbers of errors but can't see what the errors are. Appreciate any help. I'm a newbie.
Link Explorer | | pompanoartist0 -
How often the data is updating site explorer uodate his data ?
Hi guys, Im using the site explorer to compare all the seo metrics of our site to other sites.
Link Explorer | | WayneRooney
I wanted to know how often the data is updating ? I heard that its 3 month old, is that make sense ? Thanks0 -
Unable to download CSV Report
Hi, I have the email notifying me that my report is ready to download but the link simply opens a blank webpage. If I go to 'Recent CSV Reports' I can see my report listed but if I click the download button nothing happens. Can anyone suggest a solution? Thanks Hugh
Link Explorer | | HLConsulting0 -
How Is a Page Crawled by Moz When Moz Says 'No Links'?
As above, really. I've crawled a new client's site to find the Moz crawler has identified a handful of 404 errors. The Moz crawler says these pages have '0 linking domains', and OSE has no data for these pages. So how are these pages being crawled by Moz and what should I advise my client?
Link Explorer | | xerox4320