Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - help@seomoz.org
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Who uses WordPress Tags anymore?
Just curious if people are still using WordPress Tags. I wonder if with the direction Google is going the last couple years, having sites that get bloated with extraneous Tag archives just decreases the quality of the site.
Technical SEO | | WilliamBay2 -
Small blog needs paid SEO help
I looked at Moz's recommended companies but they are all for huge blogs and websites. I cannot afford those fees. My blog is small but growing and was hit by a Google Panda tweak back in July. My organic traffic disappeared overnight and although I've been working hard to improve my blog, I still get no organic traffic after being on the first page. I would like to pay someone to look at my site and give me help but I cannot pay what these huge companies are asking for. Any advice?
Technical SEO | | blogger20130 -
Adding an SEO Friendly Blog Module
Hi, Our website is developed in .NET. We need to add a new "out the box" blog module. Is there such a thing as a blog module that is good for SEO. For example updatable Page Titles and Title Tags etc. If so can anyone recommend one Thanks Andrew
Technical SEO | | Studio330 -
Duplicate Content in Wordpress.com
Hi Mozers! I have a client with a blog on wordpress.com. http://newsfromtshirts.wordpress.com/ It just had a ranking drop because of a new Panda Update, and I know it's a Dupe Content problem. There are 3900 duplicate pages, basically because there is no use of noindex or canonical tag, so archives, categories pages are totally indexed by Google. If I could install my usual SEO plugin, that would be a piece of cake, but since Wordpress.com is a closed environment I can't. How can I put a noindex into all category, archive and author peges in wordpress.com? I think this could be done by writing a nice robot.txt, but I am not sure about the syntax I shoud use to achieve that. Thank you very much, DoMiSol Rossini
Technical SEO | | DoMiSoL0 -
Should each new blog post be added to Sitemap.xml
Hello everyone, I have a website that has only static content. I have recently added a Blog to my website and I am wondering if I need to add each new Blog post to my Sitemap.xml file? Or is there another way/better way to get the Blog posting index? Any advice is greatly appreciated!
Technical SEO | | threebiz0 -
Exclude Child URLs from XML Sitemap Generator (Wordpress)
Hi all, I was recommended the XML Sitemap Generator for Wordpress by the very helpful Keith Bloemendaal and John Pring - however I can't seem to exclude child URLs. There is a section Exclude items and a subsection Exclude posts. I have tried inputting the URLs for the pages I don't want in the sitemap, however that didn't work. So I read that you have to include a list of "IDs" - not sure where on earth to find that info, tried the page name and the post= number from the URL, however neither worked. I hope somebody can point me in the right direction - and apologies, I am a Wordpress novice, and I got no answers from the Wordpress forums so turned right back to SEOmoz! Cheers.
Technical SEO | | markadoi840 -
.me vs .com for new personal blog site
Hi guys, this is my first ever post on SEOMoz (woo!) I have researched and I did see someone else ask something similar but I still wasn't clear, so i hope this question is not considered a duplicate and can go on to help other people too Enough waffle For various reasons I am moving our company blog to startup a personal blog instead and I have bought a couple of appropriate domain names in a firstname/lastname format for the new blog, basically: myname.me and iammyname.com My question is, which would you consider 'better', if either, for SEO? (bonus point: are there any other non-SEO factors I should consider?) Obviously the second name is longer, but it is a .com and I hear all the time that .com is king and .me is waaaay behind Ultimately I want to rank #1 for my name If it was your site and your blog and you had my choices which one would you go for? Many thanks for your help. I'm looking forward to being part of the SEOMoz community and learning a lot from you guys, cheers, Nick
Technical SEO | | NickDavis0 -
Properly Moving Blog from Index to its Own Page
Right now I have a website that is exclusively a blog. I want to create pages outside of the blog and move the blog to a page other than the index file e.g.) from domain.com to domain.com/blog I will have the blog post pages stay in the root directory. e.g.) domain.com/blog-post Any suggestions how to properly tell SE's and other websites that the blog has moved?
Technical SEO | | Bartell0