Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml
-
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can I analyze my site?
Hello Moz communication, please how can I analyze my site https://bettyrecipe.com/ with moz tool and also run a link audit? So that I can see spammy backlinks?
Link Explorer | | loadedroms0 -
Compare Metrics in Moz
Wondering if there are any webinars or websites available to help me better understand and break down the different categories and what they specifically mean when you run a website through the Compare Metrics in Moz?
Link Explorer | | MainstreamMktg0 -
None of the pages crawled contain an email address or links to a social profile...
I'm trying to reduce the amount of spam flags our website has using the Open Site Explorer tool. Currently, we have 2/17 flags based on: Large Site with Few Links - We found very few sites linking to this site, considering its size No Contact Info - None of the pages crawled contain an email address or links to a social profile The "few links" can be ignored, we're working on this. We don't have a visible email address on the website and we don't particularly want one. We prefer customers to fill out our online form or to call us. Moz say "none of the pages crawled contain an email address OR links to a social profile" - we do have social buttons on every page of the website, but these are official Facebook and Twitter buttons that are rendered with Javascript, so don't actually appear in the page source on load. If we replace these with actual links to our pages using Facebook and Twitter icons, will this flag be removed since Moz are saying "or links to a social profile" - making it sound optional. Thanks!
Link Explorer | | LiamMcArthur0 -
When the site explorer will recognise the new TLD?
Hey, I own a website, called tipster.website I noticed that the site explorer doesnt recognise the domain extension. When they will be added too? I dont know if the other extensions work , but this one definitely doesnt work. ps: majestic.com also don't recognise the extension.
Link Explorer | | nyanainc0 -
[No Title] in Top Pages using Open Site Explorer
Hello!
Link Explorer | | eagleleather
All of my pages contain properly formatted title tags which display in the browser. However, when I view the 'Top Pages' area of the Open Site Explorer, 99% of those pages show [No Title] in the Page Title & URL column. Does anyone know what might be causing this?0 -
Moz Crawl Canonicals and Duplicates
Hi all, I am using Moz Crawl to analyze some sites I am having to optimize.
Link Explorer | | Eurasmus.com
I keep seeing many of my pages detected as duplicate content when they have the rel=canonical applied. Example: www.spain-internship.com/zh-CN/blog-by-aaron
I have seen that in other sites. Of course I understand that Moz is not perfect but, is there a known issue or am I doing something wrong with the canonicals? Regards,0 -
Page Authority dropped to 1 for subdirectory in my site
Hey, one of my sites https://www.automationanywhere.com/testing has had all it's pages drop to a page authority of 1 in Moz and the campaign doesnt appear to be indexing any pages from this directory. We launched a new site in this directory on the 12th and havent been getting any moz love since. The pages are indexable and followable, and are being indexed by google and others. The site on the root domain has been unaffected. Please help me get my moz campaign back on track. Thanks!
Link Explorer | | aatethys0 -
Why does OSE & Moz Q&A sections not work for me in Chrome any more?
A few weeks ago I started seeing instances of OSE not showing all the information when it was showing I was logged in while using Google Chrome. It would work, then not work and kept showing me the button to sign up for Moz Analytics even though I was logged in. I'm a Pro Member, which is confusing. Then I noticed that while logged in, I would be logged out somehow when trying to contribute to the Q&A's and it would bounce me back to the Moz Home page or show me the white Login button in the upper right. Again, in Chrome only. I was told by support to clean my cookies/cache and that works for maybe a few hours, but then would resort back to logging me out. It has become very annoying, and now can only work logged in from FireFox. Not a huge issue, but everything else I do all day is in Chrome. Is anyone else having this same issues? Any recommendations? Thanks in advance. - Patrick
Link Explorer | | WhiteboardCreations0