Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - help@seomoz.org
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Faq problem in wordpress website
hi hi i add script faq in my site but doest show in serp and add moudle ithink may be problem in version wordpress or theme not stable ... my site url is https://giftmax.ir example when i search" خرید گیفت کارت" my competitor show faq script but my site doesnt show please help to resolve this problem thanks moz
Technical SEO | | ahmad21aga0 -
Extremely slow TTFB (WordPress)
Hi guys. I’m not sure if this is a relevant place to post such question, but I’m really desperate. The TTFB on my website is ridiculously slow no matter what I do. I’m using Apache and the latest version of WordPress on a VPS (NameCheap). 2 CPU Cores 2 GB RAM 40 GB SSD RAID 10 1000 GB/mo Bandwidth GZIP compression is enabled, images are optimized, CSS, JS, HTML optimized, caching enabled (WordPress plugin), CDN is enabled (Cloudflare), PHP updated to the latest version. As I said, no matter what I do, TTFB remains very slow (most of the time the TTFB of my website is between 1.5 - 3 seconds). Google Page-speed Insights score is 86 for mobile and 95 for desktop (with 1.5s TTFB). What do I do about it? Will it affect the rankings of my website? Thank you so much.
Technical SEO | | AslanBarselinov0 -
Referencing links in Articles and Blogs
Hi I am wondering if the <sup>tag in html is picked up by google as a reference point?</sup> I.e when you put a superscript in word it puts a small number next to your sentence. Then you have a list of reference at the end of the blog/article does google recognise this?
Technical SEO | | Cocoonfxmedia0 -
Wordpress Woocomerce Recommended SEO URL structure
Hi Mozzers ! Thanks for looking. I have a new shop in development (http://www.vintageheirloom.biz), I'm now using WordPress & Woocommerce. I've asked Woocommerce whether it is possible to remove the 'shop' and 'product-category' categories. They say it is, but it isn't recommended, it can slow site speed & create possible duplicate pages. I'm wondering what seasoned SEO experts opinions are on my particular structure? I've heard that a flat structure is recommended, but ecommerce shops as I understand pose their own issues, so any feedback would be appreciated.. Here's some URL examples: http://vintageheirloom.biz/shop/bags/ - this for the category bags http://vintageheirloom.biz/product-category/bags/shoulder-bags/ - this for shoulder bags a child of bags category http://vintageheirloom.biz/shop/2-55-bags/vintage-chanel-caviar-skin-2-55-bag/ - a product The last URL contains the category 2-55 bags. The products name also includes the phrases 2-55 bag. Should this level of repetition be avoided or is it best to keep the whole phrase 'vintage-chanel-caviar-skin-2-55-bag/' for SEO purposes? Thanks for any help you can give me around this issue! Kevin
Technical SEO | | well-its-1-louder0 -
Are there SEO implications to blocking foreign IP addresses?
We are dealing with a foreign company that has completely ripped off our entire site template, design and branding. This is such an inconvenience and we've had similar things happen enough in the past that we're considering blocking large ranges of IP addresses from accessing our site, via htaccess. Is this something that will potentially cause problems with search engine bots crawling or indexing our site? We are in the US and our site is hosted in the US, but I'm not sure if the major search engines could potentially be using foreign based bots. Looking for any insight on this or if there are any other potential SEO problems to consider. Thanks
Technical SEO | | ZeeCreative0 -
Adding .html To Wordpress Site
I am working on a team (my part is the SEO) where the developer added the .html extension to the permalinks. I don't understand why, on Wordpress, you would do this. Is there any benefit, or penalty for it as an SEO standpoint? I usually just set mine up %postname% as the permalink structure, but I am not the web design lead on this project. I asked the designer why, but he seems to be reluctant to answer any of my emails about his work, (like he is above that or something). Not wanting to make things worse, I dropped it and thought I would ask here since I saw a few posts in reference to it today in the forum. Is there an advantage or disadvantage (either way) to using the .html extension on a Wordpress site?
Technical SEO | | kbloemendaal0 -
Syndicated posts extracts on wordpress and impact on SERPS
On our main site (http://www.deeperblue.com) we've been syndicating posts (not the full posts just link and short extract) from a trusted partner of ours. These posts are listed as Diverwire Staff and point directly back to the original website. What i'm concerned about is the impact on SERPS - we don't want to be penalised by any of the search engines.
Technical SEO | | StephanWhelan0 -
Subdomains at Yola, Blogger, Wordpress
If the purpose of constructing a site or blog is for SEO ie a linking microsite, is it better to keep as a subdomain or to register on its own domain. The question is how much of the Domain Authority of that site will flow through the subdomain to linked site. I note that these subdomains have PA of 1, does this answer my own question?? Thanks eg widgets.yolasite.com or widgets.wordpress.com
Technical SEO | | seanmccauley0