Rogerbot did not crawl my site ! What might be the problem?
-
When I saw the new crawl for my site I wondered why there are no errors, no warning and 0 notices anymore.
Then I saw that only 1 page was crawled. There are no Error Messages or webmasters Tools also did not report anything about crawling problems.
What might be the problem?
thanks for any tips!
Holger -
I am running into this problem as well. Does anyone have an answer. I was a customer a year ago and had no problems. Stopped for a while and then came back, my site was never able to be crawled again. A few months later I did make a few changes on my site, and just tried again today, but it only gets 1 page. Once before they told it had to do with something like http 1.0/1.1 which makes no sense. Not sure why google, bing and the rest can crawl the site, but this service can't. Hopefully someone has an answer. I looked at that link that is mentioned and I do not feel that any of those are causing the issues.
-
Hi there,
Thanks for reaching out! I went ahead and started a ticket for you to do some digging. I will run a few tests on my end to see where it is causing the issue. I will get in touch you ASAP
Thanks again for reaching, hope I could talk to you soon!
Best,
-
Thanks Keri,
But its not about these common problems, because Rogerbot crawled the properly before. May be a ticked will help to find the cause.
Holger
-
Hi! Here are some common reasons why only one page is crawled. If this doesn't solve it, opening a ticket with the help desk would be the next step.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Compare sites?
I'm frustrated, so want to ask a stupid question....My site.. www.seadwellers.com outranks my biggest competitor in most Moz catagories... www.rainbowreef.us ...EXCEPT Facebook likes...(he has a ton) **And yet, rainbowreef.us outranks me in most keywords on
Moz Pro | | sdwellers
Google?! I know it's not simple...but Can anyone take a quick peek and give me any insight as to why??? ** Example "Dive Key Largo" keyword...he is #1 and I am #5...typical in the most important keywords!0 -
Problem to log into moz
Every time the moz logs me out from the account and then I can not log in. It shows on the left side my name like I am logged in and then when I want go to community suddenly I am not logged in. It offen shows 502 error. It was first doing on firefox, then I manage to log in chrome and now I had to log in private browsing.
Moz Pro | | Rebeca11 -
Rogerbot's crawl behaviour vs google spiders and other crawlers - disparate results have me confused.
I'm curious as to how accurately rogerbot replicates google's searchbot I've currently got a site which is reporting over 200 pages of duplicate/titles content in moz tools. The pages in question are all session IDs and have been blocked in the robot.txt (about 3 weeks ago), however the errors are still appearing. I've also crawled the page using screaming frog SEO spider. According to Screaming Frog, the offending pages have been blocked and are not being crawled. Webmaster tools is also reporting no crawl errors. Is there something I'm missing here? Why would I receive such different results. Which one's should I trust? Does rogerbot ignore robot.txt? Any suggestions would be appreciated.
Moz Pro | | KJDMedia0 -
Open Site Explorer and link numbers
I know this question has been asked many times in this forum but I still can't work it out. Why does this link: http://www.opensiteexplorer.org/links?page=1&site=www.bookpal.com.au&sort=page_authority&filter=&source=external&target=subdomain&group=0 Which is showing all links, external, to pages "on this sub domain" show 1,935 external links but this link: http://www.opensiteexplorer.org/links?page=1&site=www.bookpal.com.au&sort=page_authority&filter=follow&source=external&target=subdomain&group=0 which is exactly the same but this time shoing followed + 301 links, says "showing 1 - 50 external links) but won't show the total links (and I know the mouse-over on the question mark says it's won't show the total links, but I don't understand why it can't show the total links when it could show the total links when I requested to see "all links" instead of just "followed+301" links.) but it actually lists 700 links (14 pages, 50 results each page). I know the link list is limited to 25 links per domain but then it means you can NEVER know the total link count unless you download the full report. This makes using OSE to know numbers of links (internal, external, or otherwise) impossible. And if anyone uses the API, why the API (external+follow) returns 1,451 links? I'm sure it's an ongoing issue with people trying to get their head around all of this and I've never really been able to. Any insight would be much appreciated!
Moz Pro | | eatyourveggies0 -
Open Site Explorer Social Media?
Open Site Explorer always reports only a few facebook likes. If I go to the business page of the site, it has 100+ likes. This seems to happen a lot. Is this a OSE issue or is there an issue with the site clearly being defined to google as a connected facebook account? If OSE doesn't see the likes, will google? How can I fix?
Moz Pro | | JML11790 -
How often does Open Site Explorer Update?
How often does Open Site Explorer Update? Just trying to get a rough idea. Great tool btw.
Moz Pro | | seo3210 -
OSE Problems
I'm experiencing severe delays trying to export OSE data into CSV over the past couple of days. Relatively small reports I ran 2 days ago are still pending! Has anyone else noticed this? Also my campaign rankings have not updated since 24th April - anyone else? Logged with Moz Helpdesk but no reply as yet...
Moz Pro | | PeterAlexLeigh2 -
Why is there such a difference between Open Site Explorer and Domain Analysis?
When I look at my links in the Competitive Domain Analysis and I see I have 113,018 Ext followed links but Open Site Explorer shows 3,148. Why is there such a difference? In addition, is there a way to view the External Followed links? I want to see where these 113,018 links are coming from. Thanks, Josh
Moz Pro | | kdieruf0