Can't work out robots.txt issue.
-
Hi
I'm getting crawl errors that MOZ isn't able to access my robots.txt file but it seems completely fine to me? Any chance anyone can help me understand what might be the issue?
-
I would follow the advise from Roman, I tried to access the file via a few ways as well and it doesn't seem to be an issue to load it at all.
-
Ok, I made a quick test of your robot.txt file and looks fine, making HTTP status code test and shows me 200 code which is ok. Also, you need to make sure that your robot.txt file is accessible for the Moz crawler
- Remember to put your file is in the top-level directory of your web server.
- Also, check if your hosting provider is not blocking third-party crawlers (server level)
Here you can test the status code of your robot.txt file** https://httpstatus.io/ **If you're still having trouble this is the email of Moz help@moz.com
Also, you should check
Best of luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can check my site
can you please tell me what should I do with this site I trying a lot to solve its issues. the link is here https://alazharquranteaching.com/
Moz Pro | | saharali150 -
Crawl Diagnostics saids a page is linking but I can't find the link on the page.
Hi I have just got my first Crawl Diagnostics report and I have a questions. It saids that this page: http://goo.gl/8py9wj links to http://goo.gl/Uc7qKq which is a 404. I can't recognize the URL on the page which is a 404 and when searching in the code I can't find the %7Blink%7D in the URL which gives the problems. I hope you can help me to understand what triggers it 🙂
Moz Pro | | SebastianThode0 -
I have a duplicate content on my Moz crawler, but google hasn't indexed those pages: do I still need to get rid of the tags?
I received an urgent error from the Moz crawler that I have duplicate content on my site due to the tags I have. For example: http://www.1forjustice.com/graves-amendment/ The real article found here: http://www.1forjustice.com/car-accident-rental-car/ I didn't think this was a big deal, because when I looked at my GWT these pages weren't indexed (picture attached). Question: should I bother fixing this from an SEO perspective? If Google isn't indexing the pages, then am I losing link juice? 6c2kxiZ
Moz Pro | | Perenich0 -
Solving 'duplicate content' for page 2 of X for 1 blog post
Hi to all SEO wizards, For my Dutch blog google-plus-marketing.nl I'm using WordPress Genesis framework 2.0 with news theme pro 2.0 responsive theme. I love the out of the box SEO friendliness and features. One of those features is that it allows for a blog post or page to be divided into several pages. This results in MOZ signaling duplicate titles for all pages after the 1st page. Now I was thinking that a canonical url set to the first page should do the trick for me as I reason that the rank will go the the first page and the rest will not be seen as duplicates. Genesis does some good stuff on it's own and places the following meta tags in the header for the first page. All looks well and my question is about the same meta tags for the 2nd page and higher that I pasted below this one for the 1st page. Meta tags page 1 of X for blog post Meta tags page 2 of X for the same blog post Would it not be better to point the canonical url for page 2 till X to always point to the first page? In this case:
Moz Pro | | DanielMulderNL0 -
On Page Grading Not Working
I am not getting any results in on page grading. When I click on add keywords a list of keywords appears that I have previously entered. They have tick boxes next to them and you can filter them but SEO moz is not automtically creating grade reports? Any ideas? Obviously I've set it up wrong somehow.
Moz Pro | | SamCUK0 -
SeoMoz reporting 301 redirects I can't find
So I'm trying to find a bunch of redirects that SeoMoz is reporting that were created by Wordpress but I can't seem to find. I'm finding a bunch of redirects FROM pages like this: www.mysite.com/wp-content/themes/mimbo/page-name.com redirecting to www.mysite.com/page-name.com Anybody have any idea how I can track down the offending page that has a link to that url. I took a look at my templates and content with the pages and can't seem to find anything that would create it.
Moz Pro | | brandco0 -
Can't get Page Analysis for homepage - any ideas?
Using seomoz toolbar and can't get Page Analysis for homepage - it comes up blank for "page elements", "page attributes" and "link data". Any help would be great. Cheers, Don
Moz Pro | | nositeleftbehind0 -
This stuff works - but be patient
..or subtitled "This is how I did it" Managed to get clients second most important keyword to #1 on Google recently (most important is about 5 or 6, am getting there with that one) 🙂 Yes, we all know 'rankings aren't important, traffic/searches are' but you know what - when a Client sees their keyword at #1 it kinda helps getting paid. Thus, in summary, this is how I did it. I don't think there's anything earth shattering here but it might help a few out as I see so many 'where do i start' type posts. 1 - Do everything within Webmaster Tools that is humanly possible. Sitemaps, fix errors, preferred domain the lot. Spend ages here! 2 - Get your on page optimisation sorted out. Decide on THE main keyword for page, fire it into your campaign and test it. Do not give up until it's an A and you've done ALL the tips/hints (yes, even bold text on the page.) It's obvious but you have to tell a Search Engine exactly what the page is about - and give it a few hints too. 3 - Links - ahhh, good old links the bain of our lives. This site was struggling for links so i bought some - yes, the shame of it. I still think the best directories are a good starting point, spent about £400 or so. (BOTW, JoeAnt, HotVsNot etc,) Also, find low hanging fruit places for links. See where top 3 or 4 competitors are getting links from (via OSE) and get your site linked there too. These aren't always chargeable. 4 - Get rid of as many errors you can from Crawl Diagnostics (Roger) and do everything you can to ensure page(s) load as quick as possible. e.g. for images, resize them and reduce colour depth. 5 - Go over Steps 1-4 again and again AND AGAIN. I think that's about it (will add anything if i think of it), this all took about 4-5 months - not to do the SEO work but for Google (or any SE) to recognise it all, so thanks to all at SEOmoz, the blogs and this forum for all the assistance. (Just need to get Linkscape updated quicker now folks - i couldn't resist that one!)
Moz Pro | | Capote1