Can't work out robots.txt issue.
-
Hi
I'm getting crawl errors that MOZ isn't able to access my robots.txt file but it seems completely fine to me? Any chance anyone can help me understand what might be the issue?
-
I would follow the advise from Roman, I tried to access the file via a few ways as well and it doesn't seem to be an issue to load it at all.
-
Ok, I made a quick test of your robot.txt file and looks fine, making HTTP status code test and shows me 200 code which is ok. Also, you need to make sure that your robot.txt file is accessible for the Moz crawler
- Remember to put your file is in the top-level directory of your web server.
- Also, check if your hosting provider is not blocking third-party crawlers (server level)
Here you can test the status code of your robot.txt file** https://httpstatus.io/ **If you're still having trouble this is the email of Moz help@moz.com
Also, you should check
Best of luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Solving URL Too Long Issues
Moz.com is reporting that many URL's are to long, these particularly affect Product URL's where the URL is typically https://www.domainname.com/collections/category-name/products/product-name, (You guessed it we're using Shopify). However, we use Canonicals that ignore all most of the URL and are just structured https://www.domainname.com/product-name, so Google should be reading the Canonical and not the long-winded version. However, Moz cannot seem to spot this... does anyone else have this problem and how to solve so that we can satisfy the Moz.com crawl engine?
Moz Pro | | Tildenet0 -
I have 2 linking root domains on my URL. But I don't get the whole Root domain thing. So I don't understand how I can improve it?
I have 2 linking root domains on my URL. But I don't get the whole Root domain thing. So I don't understand how I can improve it? I copy and pasted this, from my Links page in my campaign because I can't seem to grasp what a root domain is: 'A higher number of good quality linking root domains improves a page's ranking potential'. Can some one explain to me what this is. As simply as possible. Here's my site www.Thumannagency.com Thanks in advance:)
Moz Pro | | MissThumann0 -
Page Rank Report says #6 in Google but I can't find the page anywhere
So SEOMoz says that I've consistently ranked #6 for a certain keyword. But when I search I'm no where to be found. I've done regular searches, incognito and some non-seomoz reports and all come up with nothing in Google. I noticed it a week or two ago, but didn't think it would continue. This is no bueno. I wouldn't be surprised if I got penalized (luckily my homepage relatively well for similar keywords), an old seo consultant used very spammy tactics. I recently removed them, but not before I started to notice that I fell off the map. Why would SEOMoz not recognize this, and continue to say I'm ranking well? The keyword is bpi building analyst the page is http://www.cleanedison.com/courses/bpi-building-analyst
Moz Pro | | CleanEdisonInc0 -
Can SEOMOZ Pro suggest me the best keyword to work for?
Hi, how SEOMOZ PRO can suggest me the best keyword to rank easily in the first position of Google's SERP? Is there a tool to find good keyword and got some ideas? Regards
Moz Pro | | jadlib0 -
When's the next Webinar?
I really love the webinars! I listen to them on long walks occasionally, but I haven't seen one in May or June. When will the next one be? Thanks! I apologize if this should have gone to the help desk...
Moz Pro | | WilliamBay0 -
Experencing page authority issues after a 301 redirect
We just completed a build of a new site and used 301 redirects to retain our page authority. In the first week all the interior pages reported a page authority of 1 after 2 or so weeks the page authority began to look more accurate but they were still not as high as the original pages. The strange thing is that when you click on the link to a page the page authority populates correctly but when the page finally finished loading the PA goes back down. Has anyone ever experienced this and if so how did you fix it? Thanks!
Moz Pro | | Jo_vortx.com0 -
Can I see search volumes against my keywords?
I love the keyword tracker tool on SEO Moz. However I can't work out how to see search volumes for each keyword. So while I know how I am ranking for each keyword.....I have to open a separate spreadsheet to see the local Google Monthly search volumes and then cross-refer the two, which is clumsy and time-consuming. Is there a way to amalgamate the two? It seems like such a major flaw that i'm sure its there, I can't work it out. Thanks all, Dan
Moz Pro | | danfk0