Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - help@seomoz.org
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Blog article cannibalizes our home page
Hello there, We're having a rather big SEO issue that I’m hoping someone here can help us with, perhaps having experienced the same thing or simply understanding what's going on. Since around June, our website's home page has lost the majority of its most important rankings. Not just dropping, but losing them entirely and all at once. We think it was self-inflicted: Almost at the same time, a blog article of ours (which we had recently updated) started ranking for almost all the same keywords. While our home page is a commercial page highlighting only our own product, the article that usurped the position is a comparison article, comparing our own solution to competitors. The reason we created that article is because we noticed a trend of Google increasingly favoring such comparison articles over dedicated product pages. But of course we didn’t plan to cannibalize our own home page with it. My question is whether anyone has experience with such a case? Is there a way to "tell"/influence Google to rank our home page again, instead of ranking that article? Thanks a lot, Pascal
Technical SEO | | Maximuxxx1 -
Moving wordpress to it's own server
Our company wants to remove wordpress from our current windows OS server at provider 1 and move it to a new server at provider 2. Godaddy handles our DNS. I would like to have it on the same domain without masking. I would like to make a DNS entry on godaddy so that our current server and our new server can use the same URL (ie sellstuff.com). But I only want the DNS to direct traffic to our current server. The goal here is to have the new server using the same URL as the old server so nothing needs to be masked once traffic is redirected with a 301 rule in the htaccess file. But no traffic outside of the 301 rule will end up going to the new server. I would then like to edit the htaccess file on our current server to redirect to the new servers IP address when someone goes to sellstuff.com/blog. Does this make since and is it possible?
Technical SEO | | larsonElectronics0 -
WordPress Duplicate Content Caused By Categories
Hello, We have a wordpress blog that has around 250 categories. Due to our platform we have a hierarchy structure for 3 separate stores. For example iPhone > Apps > Books. Placing a blog post in the books category automatically places it into iPhone and iPhone/Apps category, causing 3 instances of any blog post in this category. Is this an issue? I have seen 2 schools of thought on categories, 1 index follow and 2 noindex follow. I know some of our categories get indexed, but with so many, maybe it is better to noindex them. We also considered reducing our categories to 10 to 12 and use tags to provide the indexed site navigation as follows: Reviews (category) iPhone Book App, iPhone App Store (tags) but this seems a little redundant? Anyone want to take this on? thank you Mike
Technical SEO | | crazymikesapps10 -
Understanding Duplicate Titles in Wordpress
I have duplicate title errors in Wordpress and I cannot pinpoint the problem. I have my blog set up so that the home page of the blog has the most recent posts. In my campaign report somehow the page directory is being found and I can't find any links on my blog to those pages. the errors are on pages that are like the following www.example.com/blog/page/13/ www.example.com/blog/page/14/ I am using Yoast and I thought I had it set up correctly. The other pages have the correct title and canonical tags, but he urls ending with page do not, but the page directory is duplicating the home page title. How or where can i fix this issue?
Technical SEO | | hfranz0 -
Wordpress - Missing Meta Description and Title Elements Too Short
I have a wordpress site. When SeoMoz runs the crawl, my report shows 0 errors, but a lot of warnings in two areas: Missing Meta Description and Title Element Too Short. The only thing is, the URL's it shows under both of these categories look like this: http://www.millerhypnosisatlanta.com/author/Michael/page/4/ http://www.millerhypnosisatlanta.com/category/anxiety-panic/ http://www.millerhypnosisatlanta.com/tag/goals/ These are tags, categories, and author pages. How do I hide these from being seen by SeoMoz and Google? I mean I should do that right? Because there is no way I can add a description to these pages, or extend the title - right? Thanks in advance for your help!
Technical SEO | | DallasBonsai0 -
How do I add a blog to a subdirectory of a volusion store?
Based on what I've read so far, the best blog URL for SEO purposes is a subfolder (www.example.com/blog) rather than a subdomain (blog.example.com). I am having a lot of trouble adding a blog to our e-commerce store which is hosted by volusion. I have looked into installing Wordpress into a subdirectory but apparently Volusion does not allow the mysql/myphpadmin/database access needed to successfully do this. I have also looked into trying to publish my blog through ftp using Blogger, but they stopped allowing this a while ago. Does anyone know a way to add a blog to my volusion store's subdirectory? Thanks
Technical SEO | | AHH8880 -
Thesis Vs Yoast Wordpress SEO Plugin
Hi All. I've noticed some issues between the thesis themes and Yoast's plugin, namely, thesis doesn't recognise Yoast's plugin in for title tag and meta description. Is there a way to overide this?
Technical SEO | | PerchDigital0 -
Moving Blog to Custom Domain
Hi All - I am currently attempting to move a blog (companyname.blogspot.com) that has a 2/10 GPR to a custom domain name so that I can get my targeted keywords in the domain name. The new domain will just be www.targetedkeyword.com. There are easy instructions for doing this with the A record and CNAME changes in the Blogger help, but I am worried about losing rank and links if I do this. Can you guys help me understand what the ramifications would be, and how I can accomplish this without losing the 2/10 mojo? Thanks!
Technical SEO | | Bandicoot0