Moz crawler is not able to crawl my website
-
Hello All,
I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. "
We changed the robots.txt file and checked it . but still the issue is not resolved.
URL : https://www.khadination.shop/robots.txt
Do let me know what went wrong and wjhat needs to be done.
Any suggestion is appreciated.
Thank you.
-
Hi Harini,
Jo from the Moz help team here.
I've had a look at your site and it looks like there is something server side that is blocking our bot.
When I try to cURL your site from our internal tool I'm getting a 302 to http://127.0.0.1
https://screencast.com/t/J3hhDTCM
I'm also seeing this message in this third party tool.
"The robots.txt file does not exist on this domain (302 redirect to http://127.0.0.1)"
All this points to something server side that is initiating a 302 redirect for our bot. While your site looks fine in the browser, our bot simply can't get through.
I would recommend reaching out to your host or web developer to see if they can check how your server is treating rogerbot/1.2
You can also ask them to check the server logs to see how your server is responding to rogerbot/1.2
You'll also want to make sure you are not blocking AWS (Amazon Web Services).
Best of luck!
Jo
-
Thank you Andy . But, the problem is MOZ crawler was unable to crawl the website even though the line " Allow: / " was present in the robots.txt.
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /?color=
Disallow: /?manufacturer=
Disallow: /?filter_material-fabric=
Disallow: /?filter_color=
Disallow: /?query_type_color=
Disallow: /?filter_size=
Disallow: /?taxonomy=
Disallow: /?view_mode=
Disallow: /?query_type_material-fabric=
Disallow: /?orderby=
Disallow: /?source_id=
Disallow: /?source_tax=
Disallow: /?shop-2__trashed?
Allow: /wp-admin/admin-ajax.phpSitemap: https://www.khadination.shop/sitemap.xml
this was the previous version of robots.txt that were been used ....
-
Hi - As Andy has said, you're not allowing Moz to crawl the site.
Read up on Rogerbot here: https://moz.com/help/guides/moz-procedures/what-is-rogerbot
-
Hi there,
You forgot the most important thing. You're disallowing a lot of things but not allowing access in the first place.
Allow: /
add this on line 2 of your robots.txt file.
Good luck
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I connect my moz crawl to datastudio to list 404s in a table
Hello I currently export 404s and other 'critical' issues to a google sheet and publish them in a monthly report in data studio Is there a way to automate this process so that my monthly report is automatically populated with critical issues
Feature Requests | | Andrew-SEO0 -
Moz Pricing Needs Update
Hi I am coming from Ahrefs and Moz plans need serious update about the limits especially rank tracking. Ahrefs is providing more keywords $0.08/keyword (5000 keywords for the advanced plan) and update interval of 3 days. Moz large plan is $0.13/keyword with intervals of 1 week. Can this be looked into? Hopefully Moz can improve the limits for keywords ranking/tracking. In 2020 where many rank tracking tools provide daily update, 1 week is a bit ridiculously long. Hopefully Moz can make some improvement in this area,
Feature Requests | | EdmondHong0 -
Even canonical present in my web pages, still moz showing canonical issues.
Even canonical present in my web pages, still moz showing canonical issues. ex : rel="canonical" content="https://www.domain.com/page/">
Feature Requests | | prasad.nueve1 -
Moz is flagging my callto: links as 404s. I "fixed" with tel: & nofollow and lost 2 DA. Help?
I changed the callto: links to tel: links in the footer and nofollowed them and my domain authority dropped 2. I'm assuming this is because we lost 2.8K "external links" Advice on how to remove 404s for Phone#'s without affecting DA? Will just using the tel: format stop Moz from crawling them as 404's? It was adding links to the 404 list that look like this: www.ocp.org/missals/callto:########### I assumed it was because to the crawlers the links looked like relative links this is what our links look like now: <a <span="" class="html-tag">href</a><a <span="" class="html-tag">="</a>tel:###########" rel="nofollow" title="Call us">########### Old footer links: <a <span="" class="html-tag">href</a><a <span="" class="html-tag">="</a>callto:###########" title="Call us">###########
Feature Requests | | OCPmusic0 -
Does moz offer a site auditor tool to imbed on your website?
similar service as http://mysiteauditor.com. just want to embed a tool on our website that allows the visitor to enter their url and have a report emailed to them.
Feature Requests | | WebMarkets1 -
Discrepancy in "np" within in my Moz Report?
| Keyword |
Feature Requests | | jessential
| np - /google-structured-data-update/ | In my "New Rankings & Insights | Hill Web Creations" report, while "Google structured data update" moved up significantly, at the same time, "np - /google-structured-data-update/" moved down significantly. Any insights that would offer a deeper understanding of the discrepancy and how to pull both up? If this corresponds with Google Analytics, the "np" Acquisitions > Campaigns > Organic Keywords > "Organic Search Traffic" Report, how is Moz rendering is New Rankings & Insights?0 -
How to reschedule a missed free moz pro walk through with the expert ?
I scheduled a moz pro walk through few days back, but by mistake I chose wrong timings for the same. How Can I rescheduled my free moz pro walk through now ?
Feature Requests | | Anupmehta0 -
Do MOZ have an affiliate platform?
Do MOZ have an affiliate platform? If so, it's hiding from me, would somebody care to point me the right way? Shall I assume this is the page / they MOZ don't have a direct affiliate program? https://moz.com/partners Many Thanks
Feature Requests | | Whittie0