Can't work out robots.txt issue.
-
Hi
I'm getting crawl errors that MOZ isn't able to access my robots.txt file but it seems completely fine to me? Any chance anyone can help me understand what might be the issue?
-
I would follow the advise from Roman, I tried to access the file via a few ways as well and it doesn't seem to be an issue to load it at all.
-
Ok, I made a quick test of your robot.txt file and looks fine, making HTTP status code test and shows me 200 code which is ok. Also, you need to make sure that your robot.txt file is accessible for the Moz crawler
- Remember to put your file is in the top-level directory of your web server.
- Also, check if your hosting provider is not blocking third-party crawlers (server level)
Here you can test the status code of your robot.txt file** https://httpstatus.io/ **If you're still having trouble this is the email of Moz help@moz.com
Also, you should check
Best of luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Struggling with the principle of backlinks, all i can see for the competition is directories?
Hey everyone, I have been using the open site explorer to check the linking domains/inbound links of competition for window suppliers. I've read that directories generate pretty much no link juice, yet all i can find for pretty much all of the well ranking competition (other than few very major companies) is directory listings. This leaves me wondering whether just submiitting to 5-10 good relevant directories would help tip me above some of the competition... I've read quite a lot about generating quality backlinks but i still haven't got my head around how to actually do anything myself that would be potentially useful. All i've done so far is post to relevant high authority directories and a couple relevant forums. I have pretty much £0 budget but a decent amount of time and strong graphic design and animation skills so feel like i can put that to use but again would have no idea what to do even if i created something share worthy. I can't imagine succesfully contacting anyone and getting them to link back to me or share my content.. Thanks for any advice...
Moz Pro | | binkez320 -
MOZ Toolbar 3.0: Can't Find the Meta Description On My Page, Why?
Hey, The new MOZ toolbar is unable to identify the meta description on any of the pages for my doman www.1099pro.com. Is there any reason that someone can see why that would be? The tag should be correct on all pages. Thanks! -Mike
Moz Pro | | Stew2220 -
We're back. THANK YOU SEOMoz!
To all the Mozzers reading this, I joined SEOMoz in a panic when we were seeing the first hints of an impending ranking crisis for our company website in October, 2012. We tanked in late December for our most important keywords. Simply GONE. Holy C---! It was devastating to say the least. Thankfully, we have a lot of work due to current clients and referrals, but the phone got quiet. That's a deadly sound.15 years of comfort zone gone. I still don't know precisely why it happened, it may have been a number of factors. But I've learned a huge amount through this crisis and can honestly say that I'm glad it happened. How weird is that? But what other fire under my butt would have gained me so much curiosity, fascination, engagement and excitement with deepening my scant knowledge? A week ago, we got our first report showing us that we have now gotten back our positions on page 1 for the 10 keywords that matter. It's continuing to improve according to today's report. So now that I've gotten somewhere- and I am well aware that it may not stick and there is still more to do -it's time to shout out a great big "Thank You". Truly, I don't think I could have survived this situation without SEOMoz and all you wonderful Mozers answering my frantic questions and lending your expertise and support to my situation. It wasn't only my own questions and the responses to those that helped me through this. I have been essentially addicted to reading a vast majority of all the posted questions because it's so dang interesting. So I've been learning bucketfuls from other people's questions and responses. My own personal SEOMoz education. It's only just begun. What an incredible culture and community we have here. I feel I can say "we" now because I am beginning to feel a part of it. And I'm not going anywhere anytime soon. I love this place. Thank you!
Moz Pro | | gfiedel11 -
I'm seeing duplicate links listed in open site explorer. Can anyone explain why this might be happening?
For each link listed in Open Site Explorer, I am seeing two identical entries. You can see why this might be a problem for both workflow and data accuracy. Any ideas why this might be happening? Is there a setting or filter that I'm missing?
Moz Pro | | StephenEggett0 -
How can authority of the page be higher then authority of the domain?
In the opensite explorer most of the links that I see have a higher domain authority and lower page authority which makes sense to me because I thought that domain authority is an aggregate of its pages authority. But apperently I was wrong because sometimes I see that the page authority is greater then its domain? How could that be?
Moz Pro | | SirMax0 -
Can 2 people from our company use the SEOMoz toolbar?
Hello, I've got SEOMoz Pro Can 2 of us use the search toolbar at once, or do we need to pay twice? Thanks! Bob
Moz Pro | | BobGW0 -
Robots review
Anything in this that would have caused Rogerbot to stop indexing my site? It only saw 34 of 5000+ pages on the last pass. It had no problems seeing the whole site before. User-agent: Rogerbot Disallow: /default.aspx?*
Moz Pro | | sprynewmedia
//Keep from crawling the CMS urls default.aspx?Tabid=234. Real home page is home.aspx Disallow: /ctl/
// Keep from indexing the admin controls Disallow: ArticleAdmin
// Keep from indexing article admin page Disallow: articleadmin
// same in lower case Disallow: /images/
// Keep from indexing CMS images Disallow: captcha
// keep from indexing the captcha image which appears to be a page to crawls. general rules lacking wildcards User-agent: * Disallow: /default.aspx Disallow: /images/ Disallow: /DesktopModules/DnnForge - NewsArticles/Controls/ImageChallenge.captcha.aspx0 -
Can I change the crawl day ?
Hi All I hope there is a simple solution to this - we have a number of campaigns setup which are all crawled, and therefore updated, on different days of the week. We review these weekly and it would be much easier if they were all crawled on the same day. Is it possible to change the crawl day for some campaigns? Thanks Roy
Moz Pro | | bluelogic0