Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - help@seomoz.org
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I use a canonical tag or 301 with Wordpress posts?
Hi all, I'm trying to determine if canonical or 301 is a better way of handling an issue on my site. The Background I've got a Wordpress website where pages are in-depth reference articles and the posts are for short news blurbs. When I produce a new resource page, I also make a short post telling readers about the new resource. I use Yoast's Wordpress SEO plug in. Sometimes, Google will rank the 200 word post higher than the 2000 word resource page. I suspect that is because of the order in which they were crawled by Google, but I do not know for sure. The Question To make sure that the resource page is seen as the most important location on the site for the topic, should I use the canonical section in the Yoast plugin on the post to point to the page? Or should I wait, and after a few days (when the news blurb is off of the first page) just 301 the post to the page? Are there any link juice considerations when using the canonical option? Thanks for the help! Richard
Technical SEO | | RichardInFlorida0 -
Problems with Wordpress and Yoast SEO Plugin
Hi, I've been working with a Wordpress site and the Wordpress SEO plugin for a few months now and I've managed to get pretty decent results for some of the keywords I was targeting, however since last week I've lost all my rankings sharply and everything dropped out. This had happened once before and the problem was that the plugin was somehow stopped but not this time. I've had a look through all of the MOZ resources and I cannot find what the problem might be. Page optimization hasn't changed and the on page rankings are the same here. Everything seems to be the same except that all my top 10 ranking disappeared. I'm new to all of this and I'm still learning so I'd appreciate if anyone could help me on this. I'm up for trying any ideas you might have but I've tried almost everything I could. I've reinstalled, updated and done everything I could with the plugin, I've checked that Google is indexing the pages and it is. I've monitored for errors on the pages and critical issues, nothing major to report so I don't know anymore what to do. Thank you so much in advance for your help. M3rgAcQ paFNOlb
Technical SEO | | rodcunha0 -
Block bad crawlers
Hi! how are you? I've been working on some of my sites, and noticed that i'm getting lots of crawls by search engines that i'm not intereted in ranking well. My question is the following: do you have a list of 'bad behaved' search engines that take lots of bandwidth and don´t send much/good traffic? If so, do you know how to block them using robots.txt? Thanks for the help! Best wishes, Ariel
Technical SEO | | arielbortz0 -
Wordpress duplicate pages
I am using Wordpress and getting duplicate content Crawler error for following two pages http://edustars.yourstory.in/tag/edupristine/ http://edustars.yourstory.in/tag/education-startups/ These two are tags which take you to the same page. All the other tags/categories which take you to the same page or have same title are also throwing errors, how do i fix it?
Technical SEO | | bhanu22170 -
Best Strategy to Rank Wordpress Category
Does anyone some suggestions for best practices when trying to rank categories using Wordpress? Unless you filter it, a category provides a complete list of all posts in that category. I can add a good title and description but am not sure beyond that. Has anyone tried using a sticky post in the category and putting keyword body text in that? Lookin' for ideas.
Technical SEO | | waynekolenchuk1 -
How to write 301 redirects in WordPress
I've successfully migrated new site to new domain (www.cmsearchmarketing.com) But I cannot get 301 redirects for pages and blog posts to redirect from the old domain (www.creativemindsearchmarketing.com). And it's my understanding I need to do a 301 for each page to maintain SEO. Here's what I've tried: RewriteCond %{QUERY_STRING} ^p=975$RewriteRule ^index.php$ http://www.cmsearchmarketing.com/top-5-questions-to-ask-an-seo-firm-before-signing-up/? [R=301,L] BEGIN WordPress<ifmodule mod_rewrite.c="">RewriteEngine OnRewriteBase /RewriteCond %{REQUEST_FILENAME} !-fRewriteCond %{REQUEST_FILENAME} !-dRewriteRule . /index.php [L]</ifmodule># END WordPress #AND ALSO# Use PHP5 Single php.ini as defaultAddHandler application/x-httpd-php5s .php BEGIN WordPress<ifmodule mod_rewrite.c="">RewriteEngine OnRewriteBase /RewriteCond %{REQUEST_FILENAME} !-fRewriteCond %{REQUEST_FILENAME} !-dRewriteRule . /index.php [L]</ifmodule># END WordPress redirect 301 /top-5-questions-to-ask-an-seo-firm-before-signing-up http://www.cmsearchmarketing.com/top-5-questions-to-ask-an-seo-firm-before-signing-up/ Any suggestions would be appreciated. _Cindy P.S. Maybe some other issues are in the way: --Old site is WP-Remix theme no longer supported, and latest WP version is 2.9.1 -- Old domain (www.creativemindsearchmarketing.com) is the primary account on BlueHost …and the new domain (www.cmsearchmarketing.com) is an addon, so the new domain's directory is within root of old domain. -- in root domain of old site there are other "handler files" that also have base file rewrites, if this is an issue: name of this file in root directory is:
Technical SEO | | CeCeBar
.htaccess.addHandlerBak -FrontPage- <limit get="" post="">order deny,allowdeny from allallow from all</limit><limit put="" delete="">order deny,allowdeny from all</limit>AuthUserFile /home/creatjo7/public_html/_vti_pvt/service.pwdAuthGroupFile /home/creatjo7/public_html/_vti_pvt/service.grp# BEGIN WordPress<ifmodule mod_rewrite.c="">RewriteEngine OnRewriteBase /RewriteCond %{REQUEST_FILENAME} !-fRewriteCond %{REQUEST_FILENAME} !-dRewriteRule . /index.php [L]</ifmodule> END WordPressAuthName creativemindsearchmarketing.comIndexIgnore .htaccess /.?? *~ *# /HEADER /README /_vti0 -
Too many links on your blog?
In all of my campaigns, I have a lot of URLs with too many links on the page (defined loosely as around or over 100 links per page); these links are virtually all found on blog pages. The link count shoots up quickly when you start using things like tag clouds, showing all the tags/categories a post is in, in addition to all the cross linking thats typical of blog posts. My question is: Does this matter? Do you work to get blog pages down under that 100 link limit, or just assume most blogs are like this and move along? If you think it does matter, what strategies have you used to cut down the number of links while still keeping popular elements like tag clouds?
Technical SEO | | AdoptionHelp0 -
Subdomains at Yola, Blogger, Wordpress
If the purpose of constructing a site or blog is for SEO ie a linking microsite, is it better to keep as a subdomain or to register on its own domain. The question is how much of the Domain Authority of that site will flow through the subdomain to linked site. I note that these subdomains have PA of 1, does this answer my own question?? Thanks eg widgets.yolasite.com or widgets.wordpress.com
Technical SEO | | seanmccauley0