Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml
-
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Give a Solution my DA is not increasing and not index my backlinks in MOZ??
DA of my site is not increasing. It seems like Moz is not indexing my backlinks. Multiple backlinks of my site [ Tech Spotty ] is not indexing in Moz that is why DA is not increasing. Backlinks of my site have indexed in both Ahref and Semrush both not showing in Moz. DR has increased to 51 but DA is still 7. If anyone has the solution please let me know.
Link Explorer | | fbowable1 -
MOZ doesn't work for .dating and .chat domain extensions
I have been a MOZ subscriber for a few years now. I don't think MOZ works for .dating and .chat domain extensions. I have 2 sites that have authority 1 despite back links. Here are the details: https://oooo.dating > DA = 1 https://talk.chat > DA = 1 oooo has 221 links (Google search console) talk has 1317 links (Google search console) May be a MOZ staff member can look into this. If you are customer and use some of the newer domain extensions please share your details if you have the same problem.
Link Explorer | | dmcubed1 -
Unable to fully see full spectrum of links built on some backlink checkers
We have built various links in directories and blogs to our website. We can clearly see that these links have been established on these websites. Some we have built well over 6 months ago. However, upon using a range of tools such as Moz Open Explorer, Cognitive SEO Backlink Explorer and SEM Rush's Backlink Checker, we cannot see in any of these backlink checking tools the full spectrum of links we know we have built displayed. In instances of using these tools, we are only seeing a very small sample of links that we have built. Our question here is then threefold: Why is it that a lot of these tools don't pick up the links we know we have built? Does google eventually see all the links we have built? Or does it suffer from the same problem as backlink checkers, unable to identify all built links? Is there a backlink checker that is throughly comprehensive, or close to, in its ability to identify all links? We have heard good things about Ahrefs and would love to hear people's thoughts here.
Link Explorer | | Gavo0 -
Confused about the link metrics showing up in MOZ & OSE
For my campaign "itelligence Applications" which is for site itelligencegroup.com/uk/ - MOZ seems to be giving me different stats on links, depending on which tab I look it in the Link Analysis section. Then when I look at OSE, the numbers are totally different again, not just a little bit different. Example: In MOZ Link Analysis, on the "inbound links" tab, it says there are 129 Root Domains and 6832 Total External Inbound Links. If I then look at the Competitive Metrics screen, it says there are 46,284 external followed links and 240 total linking root domains. If I then go to OSE for the same site, it says there are 14 Root Domains and 686 total links. I have no idea why these numbers vary so much or how to even report them to my client. Am I missing something here? Any help would be gratefully received! Maureen
Link Explorer | | mfrgolfgti0 -
Discrepancy between number of links from web master tools and open site explorer.
Hi I am looking at number of link to www.bagnodesignglasgow.com on open site explorer shows as total links 24 but on WMT it says 937 ? Why are these two numbers so different ? Thanks
Link Explorer | | VIVOWeb0 -
Flush Open Site Explorer data
Hi folks Recently (about a month ago) we launched a new website. However, when I use Open Site Explorer and look at the "Top pages" tab, old URL's which no longer exist, appears in the list. Is there some way to get around this? Thanks.
Link Explorer | | jyskvin0 -
Open Site Explorer not reporting all 301 redirected links
Our site had over 2,000 root domains linking to it as reported in MOZ Open Site Explorer (and Google Webmaster Tools). We then changed the domain and made sure that 301 redirects were set up for all pages across the site. That was about 1 month ago. Open Site Explorer is now reporting less than 300 linking root domains. For the links that it is reporting the majority of these are being 301 redirected to the new URLs (some of them we changed the links directly). However the majority of the links that are being 301 redirected are not being reported. It is reporting most of the links coming from the old to the new domain. Google Webmaster Tools is reporting over 1,000 linking root domains to the new domain (it has a max of 1,000). We did notice that MOZ took some time to update the domain authority of the new domain. It was 1 for along time and it has now jumped up to 46 ( it was previously at 74). Maybe this is a time delay thing and eventually Open Site Explorer will report all of the 301 redirected links? It is a bit frustrating at the moment as we can't fully analyse the links to the site to try to focus on the high domain linking sites to get them to change the link directly. Also, If all links are being 301 redirected to the new domain should the authority not be close to where it was previously after one month? Our search traffic has dropped considerably since the launch of the new site and hasn't returned yet, so just wondering if the 301 redirected links pass on as much value as thre original direct links. Thanks, Damien
Link Explorer | | james.harris0 -
Since January 24 Open Site Explorer has not provided new data
Using OSE we've been tracking our stats and our competitors stats on a weekly basis to see how we're doing. We've noticed that since Jan 24,2014 the numbers for us and our competitors have been the EXACTLY same. What's going on with OSE? Company_01.png
Link Explorer | | EricksonCoaching0