Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - help@seomoz.org
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Wordpress Redirect Plugin Vs Manual .htaccess?
Hi everyone, I need to 301 redirect my old pages to new ones but i am confused between whether to choose plugin for this or i should manually rewrite the code on .htaccess file. Please give your suggestion and if you think i should use plugin then which one?
Technical SEO | | himanshu3019890 -
Wordpress Website + 404 Errors
Hi everyone, I like to do a bit of auditing for our clients using SEOMoz. Once client that's using a Wordpress website had reported over a couple hundred 404 errors. However, when checking out the links, all the webpages (that I've tested) loaded just fine. Does anyone know why this would be the case? I thought, perhaps, the website might have gone down when it was crawling, but I have no evidence to back this up.
Technical SEO | | ThinkShiftInc0 -
Too Many On Page Links Error On Wordpress Blog
I have a wordpress blog. I am getting an error message from SEOmoz "too many on page links" However SEOmoz is counting a full month of blogs as one page. For example-3 onpage internal links in each blog times 30 different blog article in a month is recorded as 90 on page links. Is there any mechanism to fix this on wordpress
Technical SEO | | wianno1680 -
Blog archives vs individual articles
In a client's blog, you can find each individual article pages as well as aggregate of articles per month or sometimes per day (including each entire article). The problem is that the article appears twice, once in a dedicated page (article page) and once with other articles (in the archive). Is there a specific SEO approach to this type of situation? Is there duplicate content? What page name should I give each archive (if at all), as there are quite a few? Thank you
Technical SEO | | DavidSpivac0 -
Paginated Home Page Duplicates on Wordpress Sites
A number of my websites created on WP are displaying duplicate home pages with these types of urls. http://www.example.com/page/10/ http://www.example.com/page/11/ http://www.example.com/page/12/ I found these duplicates using the site:search command. Basically, put in any number and the Home Page opens. With the above mentioned url structure. Any idea on why they are created, how they can be stopped and what kind of an impact they would have in terms of SEO and the penalty that comes with duplicate content.
Technical SEO | | AsadMemon1 -
Could somebody suggest a GOOD Wordpress XML sitemap generator?
We have been putzing around with Google XML Sitemaps Generator (a plug-in on Wordpress) for our Wordpress blog and we cannot get it to write an XML sitemap! Could somebody suggest a viable alternative that actually works? Thank you for your help! Jay
Technical SEO | | theideapeople0 -
Parked Domain blog directory not redirecting
My newly parked domain name, (our main website had to switch primary domains) is not redirecting properly and is causing our blog to be duplicate content. My 301 redirects work for everything else, but our parked domain /blog directory is not redirecting. I can type in both urls and then the blog appears on both sites. Not good. If I delete my blog .htaccess file, then it redirects fine. However, then our blog links are broken. So it has to do something with our .htaccess files. I do have a .htaccess file for our website, saying redirect everything to correct location, so i think this is interfering, but I cannot pinpoint it. this is the .htaccess file for the blog. BEGIN WordPress <ifmodule mod_rewrite.c="">RewriteEngine On
Technical SEO | | hfranz
RewriteBase /blog/
RewriteRule ^index.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /blog/index.php [L]</ifmodule> END WordPress main sites .htaccess (i am trying to pinpoint the issue here) Options +Includes
AddType text/html .htm .html
AddHandler server-parsed .htm .html
Options +FollowSymLinks RewriteEngine on RewriteBase / RewriteCond %{HTTP_HOST} ^(www.)?parkeddomain.com [NC,OR] RewriteCond %{HTTP_HOST} ^newdomain.com [NC] RewriteRule (.*) http://www.newdomain/$1 [R=301,L] RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^?]*)? RewriteRule (.*) /$1? [R=301,L] RewriteCond %{THE_REQUEST} ^.*/index.php RewriteRule ^(.*)index.php$ http://www.newdomain.com/$1 [R=301,L] RewriteCond %{THE_REQUEST} ^.*/index.htm RewriteRule ^(.*)index.htm$ http://www.newdomain/$1 [R=301,L] RedirectMatch 301 /index.php/(.*) /$1 Is there something obvious here, that does not look right?0 -
What is with WordPress Dupe issues?
Hi, Just wondering if anyone can explain for me why it seems every tag that is entered in WP blog posts on a site creates a duplicate page (identified by ROGER and friends in SEOmoz crawl)? Obviously if you can offer a solution (apart from the extremely obvious "don't use tags") I would be immensely grateful. Thanks so much,
Technical SEO | | ShaMenz0