Moz crawler is not able to crawl my website
-
Hello All,
I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. "
We changed the robots.txt file and checked it . but still the issue is not resolved.
URL : https://www.khadination.shop/robots.txt
Do let me know what went wrong and wjhat needs to be done.
Any suggestion is appreciated.
Thank you.
-
Hi Harini,
Jo from the Moz help team here.
I've had a look at your site and it looks like there is something server side that is blocking our bot.
When I try to cURL your site from our internal tool I'm getting a 302 to http://127.0.0.1
https://screencast.com/t/J3hhDTCM
I'm also seeing this message in this third party tool.
"The robots.txt file does not exist on this domain (302 redirect to http://127.0.0.1)"
All this points to something server side that is initiating a 302 redirect for our bot. While your site looks fine in the browser, our bot simply can't get through.
I would recommend reaching out to your host or web developer to see if they can check how your server is treating rogerbot/1.2
You can also ask them to check the server logs to see how your server is responding to rogerbot/1.2
You'll also want to make sure you are not blocking AWS (Amazon Web Services).
Best of luck!
Jo
-
Thank you Andy . But, the problem is MOZ crawler was unable to crawl the website even though the line " Allow: / " was present in the robots.txt.
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /?color=
Disallow: /?manufacturer=
Disallow: /?filter_material-fabric=
Disallow: /?filter_color=
Disallow: /?query_type_color=
Disallow: /?filter_size=
Disallow: /?taxonomy=
Disallow: /?view_mode=
Disallow: /?query_type_material-fabric=
Disallow: /?orderby=
Disallow: /?source_id=
Disallow: /?source_tax=
Disallow: /?shop-2__trashed?
Allow: /wp-admin/admin-ajax.phpSitemap: https://www.khadination.shop/sitemap.xml
this was the previous version of robots.txt that were been used ....
-
Hi - As Andy has said, you're not allowing Moz to crawl the site.
Read up on Rogerbot here: https://moz.com/help/guides/moz-procedures/what-is-rogerbot
-
Hi there,
You forgot the most important thing. You're disallowing a lot of things but not allowing access in the first place.
Allow: /
add this on line 2 of your robots.txt file.
Good luck
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why my site's DA and Pa Not Show Accurate By MOZ Extension.
my site has some minor problem which about DA and PA that DA and PA do not show accurate some time my site is yts subtitle, please tell me something about it.
Feature Requests | | darkwebvampire0 -
Can someone give details about Moz Pro Tool in Depth ?
Hi, I have been using some other tool for my monthly SEO activities, due to my advice of friend, thinking to move to Moz Pro. Can Someone give the answers for the questions SERP Ranking Re-freshness ( For how often it would update ) Mobile SERP Ranking ( Whether the tool shows the SERP Rankings on Mobiles ) Backlinks ( Does Backlinks Data has the Historical Index ) Alerts ( The important thing i need was to get instant alerts on Rankings, Backlinks Toxic Score, so how often i would receive those alerts
Feature Requests | | NishanthRaj0 -
Even canonical present in my web pages, still moz showing canonical issues.
Even canonical present in my web pages, still moz showing canonical issues. ex : rel="canonical" content="https://www.domain.com/page/">
Feature Requests | | prasad.nueve1 -
Does/could the MpzBar highlight your website in the SERPs?
Is anyone aware of a setting in the MozBar which would highlight your website within SERPs? Or, even more exciting, display a number on the bar when looking at a SERP, showing your best current position for that query (in real time)? If not, it'd be interesting to know your thoughts on the idea? Great for quick checks and references or would it discourage people from staying vigilant in the SERPs and keeping an eye on the competition? (Note: we do have MOZ Pro and are aware that it does something similar. I'm thinking more for a quick check of those difficult targets where you're not hitting the first couple of pages).
Feature Requests | | MSGroup0 -
Moz Crawler failing with https redirect?
Is there a way to get the Moz Crawl Test to work with HTTPS? I just got back this error: 902 : Network errors prevented crawler from contacting server for page. Site is set up with a standard 301 to redirect http to https - or at least I certainly hope it is! Rex Swain's HTTP Header Checker took shows a standard 301. Anyone else experiencing this error? btw - this is both a specific question and an opportunity for open discussion... Thanks!
Feature Requests | | seo_plus0 -
Does Moz Pro provide a back link suggestion area for a particular site?
Is there a tool with Moz Pro that offers link suggestions for specific sites?
Feature Requests | | DenisZilberberg0 -
Hi Moz community! Why are the number of high priority issues (404s) significantly different between MoZ and Google search console?
Hi Moz community! So when i ran a site crawl to analyze page issues for a site that has been in our account for some time now, it only showed 10 high priority (404) issues...yet when i logged into wordpress and installed yoast premium and connected to google search console, around 900 404s showed up....(we have not blocked off any pages to be crawled) does anyone know a reason why this happens?
Feature Requests | | rooify0 -
Can I embed a Moz report in my own admin site?
I'd like to be able to pull one of the reports that I have on my campaign dashboard on another site (preferably via iframe). Is this possible?
Feature Requests | | m4change0