Why RogerBot can't crawl site https://unplag.com
-
Hello
Please help me to solve the problem.
The on-page grader and Crawl Test are not working for Unplag.com website. Both said that they can't access the url. Yes, I've tried different variants like unplag.com, http://unplag.com
One more thing - RogerBot was disallowed in robots.txt file. I deleted it from the file a week ago so maybe moz index haven't been renewed.
-
Thank you. I'll try to solve the problem
-
The trouble is not with your robot.txt - in the server config you block rogerbot completely and serve a 400 for each request it makes..
If you have a user agent switcher plugin in your browser & change the user agent to rogerbot (rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr2-crawler-101@moz.com) - the server returns a 400 Bad Request.
Dirk
-
The logs are like this:
"GET / HTTP/1.0" 400 166 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr2-crawler-101@moz.com)" "-" - "https"
and of course sometimes rogerbot is trying to see the robots file:
"GET /robots.txt HTTP/1.1" 400 166 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr2-crawler-101@moz.com)" "-" - "https"
for me it looks like the rogerbot is disallowed in robots.txt but the file is like this https://unplag.com/robots.txt
-
thanks a lot!
-
Follow the advice from Jordan below and try to check your log files to see what the server response is when Rogerbot is trying to visit the site.
I noticed some DNS issues with your site - check http://dnscheck.pingdom.com/?domain=unplag.com - Nameservers don't seem to be ok. Also noticed that you have a 302 redirect from http -> https - while this should be 301. Probably not related to your main issue but worth checking.
-
Thanks.
The last crawl was after the robots.txt change.
And I don't see any errors in the dashboard.
-
After creating a fresh test campaign for the site, I'm still seeing a 400 response being served to rogerbot from https://unplag.com/. While I'm not able to pinpoint the exact setting that is causing the site to serve that response, I'd recommend checking your server logs to verify the response that is being served.
-
It's possible that your site hasn't been crawled yet (since you changed the robots.txt). You can see in your campaign dashboard (upper right corner) when the next crawl is scheduled.
Do you see any specific error codes on your dashboard?
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I have too many tittle tag issue for my site on moz site crawl error
I have too many tittle tag issue in site crawl error but when I checked manually for the error there is no title in source code. Please Help me to understand
Moz Bar | | Nileshaggarwal0 -
What does the external links column mean in the crawl report , thanks
Hi, Ran a report for www.dare2b.com report, and it showing 34780 external links. What does this mean Thanks Jeff
Moz Bar | | jefffox0 -
Community Discussion - What's Been Your Experience With Moz Content?
When the content developed Moz Content, I was excited as can be about having another tool in the content marketing and content strategy repertoire. I knew it could and would help marketers better identify the content they should be creating and make it easier for them to move the needle for their brands. Since it's been available, I've had fun using Moz Content, seeing it as a great vehicle for flattening the learning curve for content ideation and creation. In a recent post, Here's How I'm Using Moz Content for Mining Local Link Opportunities, David Farkas described how brands can use Moz Content to better create localized content. I'd like to know how you're using it, or if you're using it: Have you tried Moz Content? And if not, what's stopping you? If you have used it, what are you really liking? What would you change? What, if any, additional features you'd like to see added? What tips can you share for helping others get the most out of the tool? Looking forward to reading the comments below.
Moz Bar | | ronell-smith3 -
How is Moz's DA score so high?
Obviously they make the tool that measures this, but is their own score inflated because of this? I know DA isn't strictly a measure of popularity, but Moz's own score seems a little _too _high compared to some other sites... 92 - Moz.com 90 - Salesforce.com 87 - Airbnb.com 84 - Priceline.com 83 - Squarespace.com Can someone shed some light on how DA is actually measured?
Moz Bar | | WickVideo0 -
Moz's on-page grader
Hello, I am having an issue with Moz's on-page grader. Grading the page, http://www.babyklar.dk/tigerspring/26-ugers-springet.html for the keyword 'Tigerspring 26 uger' I am receiving the error message: 'Exact keyword is used in page title' - tagging it as a highly important but easy fix. But I only have the words "Tigerspring 26 uger" in the title, so how can this not grade as Exact keyword? Is there somthing wrong with the system? Or is it me? Thanks very much. Best regards
Moz Bar | | Babyklar
Martin0 -
How much time should I wait between Crawl Tests?
Hello! I ask because it has happened before (and again this morning) that after doing a crawl test and repairing my site per the errors found in Moz's crawl test it still finds the same error. Even though I fixed them. Typically I do a re-crawl 6 hours after or the next day and I find the same errors. I know they are fixed because a couple of days go by and finally Moz gets it right. I had understood that the crawl test was an "on-demand" crawl of sorts, granted with limit of 2 a day. But it seems that if you re-crawl your site within a day the same results yield? It's frustrating. Is this correct? Thank you!
Moz Bar | | md30 -
Internal Links Count in Crawl Report
My understanding of the 'Internal Links' results in a moz crawl report is that it represents the number of links on the given page that link to other pages on the same site.Assuming this is a correct assumption: We recently ran a crawl report on www.phase1tech.com. Some of the pages are coming back with a large amount of 'internal links'. These 2 pages for example are showing 800 internal links: http://www.phase1tech.com/Upcoming-Events
Moz Bar | | AISEO
http://www.phase1tech.com/Contact Then there are a number of pages coming back with 705 Internal Links, including: http://www.phase1tech.com/Dalsa-CameraLink-Cameras
http://www.phase1tech.com/Hitachi-CameraLink-Cameras At best there are approximately 70-80 links on these pages. Where are these large counts coming from? Is there a means to see what the links being reported on are? At the same time the 'Too Many On-Page Links' indicates 'No' for some pages with a high number of links, and 'Yes' for pages with a low number of links. For example: http://www.phase1tech.com/Baumer-SX-Series
Too Many On-Page Links: Yes
Internal Links: 2
What's up with that?0 -
Where's Rogerbot?
Could someone please tell me where Rogerbot lives now?! Unless I'm having distorted memories, I used to be able to crawl websites with Rogerbot (that are not set-up as Campaigns). Could someone please let me know where to find this now? p.s. I didn't really want to Q&A this, but after a while clicking around moz I'm now even questioning myself!
Moz Bar | | GregDixson0