Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - help@seomoz.org
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Some Issues about my Blog
I am facing issue regarding to my Blog https://digitalmedialine.com/blog/. As some pages are not Rank in google yet. Can Anyone help me out how to rank those blogs to improve my Traffic. Thanks in Advance.
Technical SEO | | qwaswd0 -
Moving Blog and 301 Redirect Advice
Hello Moz Community, We recently moved our blog from its own domain to a directory on our website. We do not plan on moving over all the old blog posts because a majority most of them are based on events or time-sensitive information that has passed. We need advice on what to do with all of the old blog URL's? Should we just 301 all of them to the new blog directory on our website (www.domain.com/blog)? Should we take the time to move over all the old blog content and put the appropriate 301's in place? Any and all advice would be greatly appreciated. Thank you in advance.
Technical SEO | | All-Star-Vacation-Homes
Best,
Rich0 -
Wordpress Redirect Plugin Vs Manual .htaccess?
Hi everyone, I need to 301 redirect my old pages to new ones but i am confused between whether to choose plugin for this or i should manually rewrite the code on .htaccess file. Please give your suggestion and if you think i should use plugin then which one?
Technical SEO | | himanshu3019890 -
Multi language blog
Does google consider it to be spam by having multiple languages posts on a blog such as English, Arabic, spanish blog posts.
Technical SEO | | MozAddict0 -
Blocked by robots
my client GWT has a number of notices for "blocked by meta-robots" - these are all either blog posts/categories/or tags his former seo told him this: "We've activated following settings: Use noindex for Categories Use noindex for Archives Use noindex for Tag Archives to reduce keyword stuffing & duplicate post tags
Technical SEO | | Ezpro9
Disabling all 3 noindex settings above may remove google blocks but also will send too many similar tags, post archives/category. " is this guy correct? what would be the problem with indexing these? am i correct in thinking they should be indexed? thanks0 -
Noindex, nofollow on a blog since 2009
Just reviewed a WordPress blog that was launched in 2009 but somehow the privacy setting was to not index it, so all this time there's been a noindex, nofollow meta tag in the header. The client couldn't figure out why masses of content wasn't showing up in search results. I've fixed the setting and assume Google will spider in short order; the blog is a subdirectory of their main site. My question is whether there is anything else I can or should do. Can Google recognize the age of the content, or that it once had a noindex meta tag? Will it "date" the blog as of today? Has the client lost out on untold benefits from the long history of content creation? I imagine that link juice from any backlinks to the blog will now flow back to the main site; think that's true? Just curious what others might think of this scenario and whether any other action is warranted.
Technical SEO | | vickim0 -
Thesis Vs Yoast Wordpress SEO Plugin
Hi All. I've noticed some issues between the thesis themes and Yoast's plugin, namely, thesis doesn't recognise Yoast's plugin in for title tag and meta description. Is there a way to overide this?
Technical SEO | | PerchDigital0 -
When to SEO optimize a blog post?
Hi there, Here's our situation: there are two people working on the blog. person 1) writes the posts person 2) SEO optimizes the posts I know this is not ideal but it's the best we can do and it's a whole lot better than no blog. 🙂 I'm the fellow optimizing the posts. I've found that my best SEO efforts usually slightly undermine the readability of these posts -- not in an extreme way, I'm not going overboard with keywords or anything. Rather, things like a sexy & enticing article heading may have to be dummed down for search engines... Because of this dumming down, I like to wait a couple of weeks to SEO optimize our posts, the logic being that we get the best of both worlds: a happy regular readership on topic articles that are clearly described for (and aligned to the terms used by) our search engine visitors What I'm wondering is, Generally: can you see any problems with this setup? would you do it differently? Specifically: does Google (et al) punish this sort of backwards re-writing? and, does it somehow amount to less SEO mojo when done retroactively? Thanks so much for your time! Best, Jon
Technical SEO | | JonAmar0