Moz crawler is not able to crawl my website
-
Hello All,
I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. "
We changed the robots.txt file and checked it . but still the issue is not resolved.
URL : https://www.khadination.shop/robots.txt
Do let me know what went wrong and wjhat needs to be done.
Any suggestion is appreciated.
Thank you.
-
Hi there! Tawny from Moz's Help Team here!
I think I can help you figure out what's going on with your robots.txt file. First things first: we're not starting at the robots.txt URL you list. Our crawler always starts from your Campaign URL and goes from there, and it can't start at an HTTPS URL, so it starts at the HTTP version and crawls from there. So, the robots.txt file we're having trouble accessing is khadination.shop/robots.txt.
I ran a couple of tests, and it looks like this robots.txt file might be inaccessible from AWS (Amazon Web Services). When I tried to curl your robots.txt file from AWS I got a 302 temporary redirect error (https://www.screencast.com/t/jy4MkDZQNbQ), and when I ran it through hurl.it, which also runs on AWS, it returned an internal server error (https://www.screencast.com/t/mawknIyaMn).
One more thing — it looks like you have a wildcard character ( * ) for the user-agent as the first line in this robots.txt file. Best practices indicate that you should put all your specific user-agent disallow commands before a wildcard user-agent; otherwise those specific crawlers will stop reading your robots.txt file after the wildcard user-agent line, since they'll assume that those rules apply to them.
I think if you fix up those things, we should be able to access your robots.txt and crawl your site!
If you still have questions or run into more trouble, shoot us a note at help@moz.com and we'll do everything we can to help you sort everything out.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are there any plans for a Moz connector to Google Data Studio
Hello all, As title reads, are there any plans for a Moz connector to Google Data Studio to help automate reporting? I'm surprised this doesn't exist already because Data Studio is so heavily used. Thanks in advance. Simon
Feature Requests | | parkersj9 -
Can someone give details about Moz Pro Tool in Depth ?
Hi, I have been using some other tool for my monthly SEO activities, due to my advice of friend, thinking to move to Moz Pro. Can Someone give the answers for the questions SERP Ranking Re-freshness ( For how often it would update ) Mobile SERP Ranking ( Whether the tool shows the SERP Rankings on Mobiles ) Backlinks ( Does Backlinks Data has the Historical Index ) Alerts ( The important thing i need was to get instant alerts on Rankings, Backlinks Toxic Score, so how often i would receive those alerts
Feature Requests | | NishanthRaj0 -
Moz's New Video Ads on YouTube
I've enjoyed Moz's new video ads and want to share them with colleagues, however am unable to find the vids anywhere other than when they're delivered before a video plays on YouTube. Are they available on demand? If anyone is unfamiliar with the videos I speak of, they are new ads for Moz products like Keyword Explorer, Mozbar, and MozCon. Thanks in advance!
Feature Requests | | Choice0 -
How to solve Moz crawl error?
I want to crawl one of my eCommerce website. In my google webmaster account, there is not crawl error. But when I want to check crawl by Moz it shows error. I have made this ecommerce site with prestashop. Google still rank up my loacal keywords. Can anybody suggest me by checking Moz crawler error by checking my online shop.
Feature Requests | | Shamimgypsy0 -
How to reschedule a missed free moz pro walk through with the expert ?
I scheduled a moz pro walk through few days back, but by mistake I chose wrong timings for the same. How Can I rescheduled my free moz pro walk through now ?
Feature Requests | | Anupmehta0 -
Way to track public's use of keywords over time in MOZ?
I apologize for the basic question but is there a way to track within MOZ the public's use of my keywords over time? My traffic understandably goes down certain times of the year -- holidays, vacation season, back to school, etc. What I want to see is whether my share of the traffic available is staying steady, slipping or even increasing. Sometimes I'll see a big spike in traffic that I suspect is related more to a general increase in traffic in my keywords than anything else. Thanks for any help.
Feature Requests | | NCCompLawyer0 -
Can I embed a Moz report in my own admin site?
I'd like to be able to pull one of the reports that I have on my campaign dashboard on another site (preferably via iframe). Is this possible?
Feature Requests | | m4change0 -
Is there a way to add custom dashboards into Moz's "Custom Reporting"?
I have dashboards set up in Google Analytics that track conversions and year over year traffic data and just wanted to see if their was a way to add those extra metrics Moz's Custom Report? Any help or suggestions would be much appreciated!
Feature Requests | | Gruenagency0