Blocked By Meta Robots
-
Hi
I logged in the other day to find that over night I received 8347 notices saying certain pages are being kept out of the search engine indexes by meta-robots.
I have not changed my robots.txt in years and I certainly didn't block Google from visiting those pages.
Is this a fault on Roger Mozbot behalf? Or is there a bot preventing 8000+ pages being indexed?
Is there a way to find out what meta-robot is doing this and where? And how I can get rid of it?
I usually rank between #3 and #5 for the term 'sex toys' on google.com.au, but I now rank #7 to #9 so it would seem some of my pages/content is being blocked.
My website is www.theloveshop.com THIS IS AN ADULT TOYS SITE.
There is no porn videos or anything like that on it, but just in case you don't wish to look at sex toys or are around kids I thought I would mention it.
Blake
-
Hi Nakul
Thank you very much for the suggestions and I will start checking the code on those pages today. Thanks for heads up on the .com.au site as well I will be changing the hosting today to Aus, and thanks to you that site may get pushed up the rankings for a highly competitive term that I have been fighting to get for years
As for following my competitors links, I have been doing that for awhile and it would seem most of them either buy their links or have them on pages that are irrelevant to our niche, one of my competitors has even acquired more links than SEOmoz.com but has no were near the content. In saying that I am by no means a pure white hat SEO, more of a grey hat if such a thing exists, simply because I find it hard to get links to an adult site without it being spammy of some sorts. So I have started writing articles and linking to products mentioned, but trying to make the articles interesting/funny at the same time. Hopefully soon I will start to see some improvement.
Again thank you for your help and I will let you know how I go with those pages blocked by bots.
Blake
-
The meta robots block sounds like it's a block from your CMS or something in your header include file or something. I would check on the pages that show as blocked to see if there's any noindex sort of code in those pages. Other then that, shift in rankings could be based on your competition optimizing for the same set of keywords. I would recommend to spend some time analyzing the backlinks of the competition and see if you can gain some of those kinds of links.
On a side note: It also looks like you own both the .com and the .com.au site. The .com is hosted in AU and the .com.au is hosted in USA. Did you intend to do it the other way around ? I would imagine your rankings for the .com.au get better if it was hosted in Australia instead of USA.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Branded keyword results meta description is different then long tail branded search term in SERP
Search "Venturize", the first result is Venturize with the
On-Page Optimization | | WWWade
description "Get help on finding the right loan for your small business
and learn what to look for when comparing your options." However, if you Google "Venturize opportunity finance network", the
first result is Venturize with the description "Playing:360p@30. Embed
size: CDN:fastly. Open link. Twitter · Facebook · About Venturize ·
Contact · About OFN · Mission-Driven Lenders: Locator Map." Why is Google not taking the meta description in the longer tail search? I have checked: All of google's indexed pages for the website- A 3rd party crawl of the website- Multiple other keyword combinations Nothing produced the same result or indicated a reason why "Venturize opportunity finance network" has the incorrect meta description in the SERP.any ideas?0 -
Google finding my meta descriptions
I recently had my site redone (about a year ago). Since then google has not been using my meta descriptions much and more so using the descriptions within my site. Is there a reason for this? An example would be http://www.waikoloavacationrentals.com/vrp/unit/kolea-14/
On-Page Optimization | | RobDalton0 -
Can I robots.txt an entire site to get rid of Duplicate content?
I am in the process of implementing Zendesk and will have two separate Zendesk sites with the same content to serve two separate user groups (for the same product-- B2B and B2C). Zendesk does not allow me the option to changed canonicals (nor meta tags). If I robots.txt one of the Zendesk sites, will that cover me for duplicate content with Google? Is that a good option? Is there a better option. I will also have to change some of the canonicals on my site (mysite.com) to use the zendesk canonicals (zendesk.mysite.com) to avoid duplicate content. Will I lose ranking by changing the established page canonicals on my site go to the new subdomain (only option offered through Zendesk)? Thank you.
On-Page Optimization | | RoxBrock0 -
Meta tags
What is the best way to create unique meta tags for pages like http://www.gobol.in/brands/a http://www.gobol.in/brands/b http://www.gobol.in/brands/c http://www.gobol.in/brands/d
On-Page Optimization | | Obbserv0 -
Meta Title for Category Pages
I am trying to improve the SEO for a few key category pages on my blog. I have two ranking really well so I want to boost them and bring up some others. Can I put two keywords in the meta title? Example: Maternity Deals | Pregnancy Deals Or should I just stick with one? Thanks!
On-Page Optimization | | dealblogger0 -
Regarding the META KEYWORDS tag
Two questions regarding the META KEYWORDS tag: Does Google actually penalize you for using the KEYWORD META tag, or does it just ignore it? On a related matter, I've noticed that when I use the On-Page Optimization Tool, it indicates that my header includes the KEYWORD tag, although I've taken great pains to remove keywords from pages that had been optimized by a previous SEO. I'm using Magento - Could it be a legacy issue? Would that make a difference?
On-Page Optimization | | RScime250 -
Meta Description not displaying in Google
Hi Mozzers, I have a client that wants to change the way the meta description for some of his pages is being displayed. I've tried using the NOOPD and NOYDIR tags and its not worked. This isn't the client but perform this search in Google.ie - "accommodation newry daft" you get this result - http://www.google.ie/#hl=en&sclient=psy-ab&q=accommodation+newry+daft&pbx=1&oq=accommodation+newry+daft&aq=f&aqi=&aql=&gs_sm=e&gs_upl=11197l11712l2l12016l5l5l0l0l0l0l186l851l0.5l5l0&bav=on.2,or.r_gc.r_pw.r_qf.,cf.osb&fp=f5c640577bb5a285&biw=1600&bih=775 See how Daft.com (2nd results down) has the text "10+ items" in the description- my client has this as well as do many other competitors but its not present in the meta description tag. Anyone know how to get rid of this and get the good old meta descrition in the SERPs? Thanks BUsh
On-Page Optimization | | Bush_JSM0 -
How to Define Best Structure of Title Tag and Meta Description?
I want to define best structure of Title Tag and Meta Description for my eCommerce website. There are too many categories over there and many be increase in near future. There are 7K+ products live on website so, it's quite hard to develop Title Tag and Meta Description for all product pages. That's why I have set automatic Title Tag and Meta Description creation with help of developers. I want to do similar stuff for my category pages. Can anyone suggest me structure of Title Tag and Meta Description which suppose to help me in well over Google's organic result. I have define structure as follow. Title Tag: Category Name: FREE Shipping & Discount Category Name | Vista Stores Meta Description: Category Name with FREE shipping, 100% price guarantee, Buy and save 20% to 40% on attractive, modern and discount Category Name. Meta Keywords: Category Name I also want to give real time example with specific URL. (http://www.vistastores.com/69_21/outdoor/patio-umbrellas.html) Category Name = Patio Umbrellas so with help of above structure Title Tag and Meta Description will be as follow. Title Tag: Patio Umbrellas: FREE Shipping & Discount Patio Umbrellas | Vista Stores Meta Description: Patio Umbrellas with FREE shipping, 100% price guarantee, Buy and save 20% to 40% on attractive, modern and discount Patio Umbrellas. Meta Keywords: Patio Umbrellas So, Does it matter for good performance? If any inputs in same direction so I am eager to read it.
On-Page Optimization | | CommercePundit0