Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml
-
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Why am I not seeing search.yahoo.com in my site's backlink profile?
When looking at the backlinks for a client's competitor I found that they had a backlink on search.yahoo.com/reviews and our domain (harmonyhit.com) did not. I was using the Link Explorer tool. However, when I go to yahoo Search I see that the website of harmonyhit.com is indexed in Yahoo search and has links to the site. Also, it's sitemap has been recently crawled in Bing webmaster tools which Yahoo uses. Why does our client have the backlink accounted for and we do not?
Link Explorer | | DigitalThirdCoast0 -
Moz DA is not improving my site DA why?
Hello Moz community, I have created a lot of high domain authority backlinks for my official service website but my DA still less than 15, why my site DA does not increase as I have created huge backlinks.
Link Explorer | | Ijiubu7680 -
Moz isn't crawling all my backlinks.
Moz isn't crawling all my backlinks. It's showing only 29 referring domain when I have more than 200 referring domains linking to my website. My website URL is 360gisthub.com.ng 360gisthub.com.ng
Link Explorer | | Mustybay0 -
My company has about 70 live pages. Moz crawled 6k. Why?
My company has about 70 live, published pages on our website, which is managed through HubSpot. When Moz crawled the URL, it found 6K+. I noticed several were URLs with a campaign tag. Is this normal? It seems excessive to me. If there's a problem, how do I fix it?
Link Explorer | | ActionableResearch1 -
Top Pages are all 301
Hi, New to Moz, so sorry if this is a n00b question. I can see on the 'Top Pages' page, that the HTTP status of all my Top Pages is 301, which means they have been permanently redirected. I am trying to find out what they have been redirected from / to so that I can clean them up and make then nice 200s, but I can't see the info anywhere? 😕
Link Explorer | | plainsailing0 -
Why aren't my "page social metrics" increasing?
I post a lot to Facebook & twitter, & my "page social metrics" haven't budged in 4 weeks. I even stopped using bit.ly & stated the full URL as a test. Still no change. The fb account is /getgoodgifts & twitter is /giftsing. Thoughts on why social metrics aren't increasing?
Link Explorer | | giftsing0 -
Open Site Explorer -> Just Discovered -> Internal Links: Only shown 1 url link per day
Hi, We have a page that have like 10-15 new content/pages per day.
Link Explorer | | Inspiringinterns
eg. domain.com/post-id-01/, omain.com/post-id-02/, omain.com/post-id-03/ etc
But somehow, under Just-Discovered, internal links only show 1 per day when it suppose to display like 10-15.
The layout for each pages is exact same except for the content changes and the url's ID changes. Our competitor on the other hand, they have like 5 new page per day, but somehow Moz shows their internal links as 25 per days. Each of their new url have 4 more prefix number.
Eg:
new-page-01?track=990
new-page-01?track=991
new-page-01?track=992 etc
The above example url is technically as 1 page if without the "?" affix. Whiles all our new pages/url have unique id, but Moz only show 1 a day. Please advise on how to get Moz to shows all the new pages in Just-Discovered (internal link). Thanks, Cam0 -
When the site explorer will recognise the new TLD?
Hey, I own a website, called tipster.website I noticed that the site explorer doesnt recognise the domain extension. When they will be added too? I dont know if the other extensions work , but this one definitely doesnt work. ps: majestic.com also don't recognise the extension.
Link Explorer | | nyanainc0