Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml
-
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Solved Moz Link Explorer slow to find external links
I have a site with 48 linking domains and 200 total links showing in Google Search Console. These are legit and good quality links. Since creating a campaign 2 months ago, Moz link explorer for the same site only shows me 2 linking domains and 3 total links. I realise Moz cannot crawl with the same speed and depth as Google but this is poor performance for a premium product and doesn't remotely reflect the link profile of the domain. Is there a way to submit a sitemap or list of links to Moz for the purpose of crawling and adding to Link Explorer?
Link Explorer | | mathewphotohound0 -
WP Events Calendar Creates URLs Too Long in Site Crawler
My travel/tourism site is on WP and using an Events plugin that ads a calendar of events to many pages. The MOZ crawler is indexing almost 46K links with a URL too long, but the site only has about 3.8K pages indexed in Google. I can tell MOZ is indexing the same pages over and over again but just adding a random calendar month and year. Here are some examples. https://www.visitcurrituck.com/four-day-stay/?full=1&long_events=1&country[0]=US&ajaxCalendar=1&mo=10&yr=2003 https://www.visitcurrituck.com/four-day-stay/?full=1&long_events=1&country%5B0%5D=US&ajaxCalendar=1&mo=10&yr=2034 https://www.visitcurrituck.com/beach-houses-family-time/?full=1&long_events=1&country%5B0%5D=US&ajaxCalendar=1&mo=1&yr=1873 Any advice on how to prevent MOZ from indexing this way? I don't believe that Google is seeing this also, but maybe they are. I just know my site has over 63K issues and I'm sure at least 75% or more is because of the way they are picking up on the events calendar. Thanks!
Link Explorer | | CinivaAgency1 -
Doubt Regarding Moz Spam Score
My website is https://www.whatiscbd.com/ . When I check spam score of whatiscbd.com, moz show that its spam score is 9. And when i search spam score of www.whatiscbd.com, spam score is 0. Although both are same website. Why i am getting such issue. Is there any issue in my website's URL redirection. If yes then what's the issue?
Link Explorer | | HuptechWebseo0 -
403 errors in Moz but not in Google Search Console
Hello, Moz is showing that one of the sites I manage has about ten 403 errors on main pages, including the home page. But when I go to Google Search Console, I'm not getting any 403 errors. I don't know too much about this site (I handle the SEO for a few sites as a contractor for a digital marketing agency), but I can see that it's a WordPress site (I'm not sure if that's relevant). Can I assume this a Moz issue only? Thanks, Susannah Noel
Link Explorer | | SusannahK.Noel0 -
Open Site Explorer Problem
Have been trying to use Open Site Explorer this morning and just returning an error message "There was an error getting your data" or the cog whirring round for ages and not doing anything. Is it a general problem or is it just my PC?
Link Explorer | | seoman100 -
Learn how to use Open Site Explorer's Just-Discovered Links report to build links and manage your brand. Get your Daily SEO Fix!
The Just-Discovered Links report in Open Site Explorer helps you discover recently created links--within an hour of them being published. It's pretty nifty. In today's Daily SEO Fix, Nick shows you how to use the report to view who is linking to you, how they're doing it, and what they are saying so you can capitalize on link opportunities while they're still fresh plus, monitor and participate in the conversations happening about your brand. Watch our tutorial Using Open Site Explorer to Find Fresh Links and Manage Your Brand Online and get ready to dive in! This video is part of The Moz Daily SEO Fix tutorial series--Moz tool tips and tricks in under 2 minutes. To watch all of our videos so far, and to subscribe to future ones, make sure to visit the Daily SEO Fix channel on YouTube.
Link Explorer | | kellyjcoop3 -
Open Site Explorer and Expired Links?
Hi guys, I've been checking a few domains for SEO purposes and it appears that OSE is indexing a lot of links which no longer exist, this has been the case for over 6 months - does OSE cleanse expired links? As I'm sure they will be contributing to authority in the eyes of Moz?
Link Explorer | | lotterybase0 -
Why does the number of the total external links and the followed linking root domains between Open site explorer and the campaign in Moz pro doesn't match?
Opensiteexplorer returns 52.473 total external links and 715 followed linking root domains while my campaign in Moz analytics return 35.899 total external links and 1092 followed linking root domains. Does anyone knows how this is possible?
Link Explorer | | ConversionMobstars0