Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml
-
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
MOZ DA IS NOT INCRAESING
Hey my website on business cards its DA is not increasing. is there any way to increase its DA.
Link Explorer | | cartwrightkar0 -
How Do You Deal With Duplicate Content On A Retail Site
Hi Guys We make custom portfolios and boxes and every time we have a Moz site crawl, Rogerbot always returns a number of Duplicate Content Issues relating to different products. As per the image below (which I hope is visible?!) Rogerbot has flagged duplicate content onto one product that relates to 5 other different products. For instance there is duplicate content for an A4 Leather Portfolio and an A3 Leather Portfolio and an 11"x17" Leather Portfolio. I can't redirect or canonicalise to just the A4 Portfolio as they are all individually different products. The information on each page although similar, is relevant to each of the products, so rewriting a different blurb on each product page, will not be user friendly for our customers. I could ignore the duplicate content issues, but then that isn't good practise (and also makes for a very unsatisfactory looking Dashboard!) Any ideas?? Nick wzQLgBl
Link Explorer | | nick_HandCo1 -
Site has DA of 26 but no backlinks
I'm asking this for a guy over at SEOchat: http://forums.seochat.com/new-user-seo-questions-answers-123/unusual-change-domain-authority-486962.html Here's what his post says: Exclamation Unusual Change in Domain authority
Link Explorer | | ThomasHarvey
Guys and Girls, I'm having a weird issue in one of my sites , thing is the site had DA 11 in July and however when check on September Site is having 26 without me doing any type of link building and there are no new posts for this month as well. When checked using open site explorer there are no backlinks indexed in their Mozcape index as well. Site: http://www.beijingtravelchannel.com/ Can you give me any plausible reasons for this to happen, so that i can check them out. Thanks in Advance.0 -
How to force moz to crawl my backlinks?
I have some good number number of backlinks in my webmaster tools. But, open site explorer is showing very few backlinks. How to force moz to crawl all the backlinks? Or is there any way to submit backlinks to moz?
Link Explorer | | sankar7890 -
Is there some way to tell the Moz crawler not to crawl URL's with particular dynamic tags such as "?redirect-to:http//" ?
We are encountering an issue where the crawler is finding a ton of pages from our wordpress login url that has this dynamic tag in it to kinds of different blog entries. It's madness. I can't figure out what is causing these URLs to generate to be crawled in the first place! Does this sound familiar to anyone out there, any constructive suggestions? Robots text or maybe meta robots tags that would resolve this crawl issue?
Link Explorer | | RegistrarCorp0 -
Drastic Monthly Fluctuations in Page Link Metrics
We have experienced very drastic changes in our root domain numbers and as a result, have seen an odd impact on our DA. The data does not seem at all reliable at all. We went one month with over 50 recorded root domains and the next month that dropped over 75%. It doesn't make sense to be paying for a monthly pro account when the data is so clearly unreliable. What is going on? Looking for a good answer before closing our account!
Link Explorer | | TVape0 -
The difference between the statistics MOZ & Ahrefs
Hello dear. I have a website for a year. I made 10K links in one year(from 45 Root Domains).the Ahrefs.com has show full list of link root domains and pages. the ahrefs rank is 49. but MOZ said my Established Links are 5! So I have some questions, please help me: 1- what is difference between MOZ rank and ahrefs rank? 2-When Just-Discovered Link Calculate for my site? 3-why my moz trust is 0.00!?How to improve it? URL: https://esta-register.org
Link Explorer | | allyunit0 -
Open Site Explorer Only showing 2 Internal Links
Hi, for some reason Open Site Explorer only shows 2 internal links linking to our homepage (and other pages). When looking at a competitors site, they have thousands of internal links to their homepage! Are there any reasons why OSE is only showing 2 for our site, when in fact there are hundreds and thousands? Thanks.
Link Explorer | | Jaybeamer0