Blocked By Meta Robots
-
Hi
I logged in the other day to find that over night I received 8347 notices saying certain pages are being kept out of the search engine indexes by meta-robots.
I have not changed my robots.txt in years and I certainly didn't block Google from visiting those pages.
Is this a fault on Roger Mozbot behalf? Or is there a bot preventing 8000+ pages being indexed?
Is there a way to find out what meta-robot is doing this and where? And how I can get rid of it?
I usually rank between #3 and #5 for the term 'sex toys' on google.com.au, but I now rank #7 to #9 so it would seem some of my pages/content is being blocked.
My website is www.theloveshop.com THIS IS AN ADULT TOYS SITE.
There is no porn videos or anything like that on it, but just in case you don't wish to look at sex toys or are around kids I thought I would mention it.
Blake
-
Hi Nakul
Thank you very much for the suggestions and I will start checking the code on those pages today. Thanks for heads up on the .com.au site as well I will be changing the hosting today to Aus, and thanks to you that site may get pushed up the rankings for a highly competitive term that I have been fighting to get for years
As for following my competitors links, I have been doing that for awhile and it would seem most of them either buy their links or have them on pages that are irrelevant to our niche, one of my competitors has even acquired more links than SEOmoz.com but has no were near the content. In saying that I am by no means a pure white hat SEO, more of a grey hat if such a thing exists, simply because I find it hard to get links to an adult site without it being spammy of some sorts. So I have started writing articles and linking to products mentioned, but trying to make the articles interesting/funny at the same time. Hopefully soon I will start to see some improvement.
Again thank you for your help and I will let you know how I go with those pages blocked by bots.
Blake
-
The meta robots block sounds like it's a block from your CMS or something in your header include file or something. I would check on the pages that show as blocked to see if there's any noindex sort of code in those pages. Other then that, shift in rankings could be based on your competition optimizing for the same set of keywords. I would recommend to spend some time analyzing the backlinks of the competition and see if you can gain some of those kinds of links.
On a side note: It also looks like you own both the .com and the .com.au site. The .com is hosted in AU and the .com.au is hosted in USA. Did you intend to do it the other way around ? I would imagine your rankings for the .com.au get better if it was hosted in Australia instead of USA.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Writing unique meta titles for canonicalised products or not?
Hi all, I have a question (duh): I have like 10 colours of the same product. I will make one colour the canonical, do I still need to write unique meta titles for the others? I am almost certain it doesn't add value at all, but wanted to double check since I couldn't find it anywhere. Thanks a lot in advance and have a great day!
On-Page Optimization | | TheOnlineWarp0 -
Magento Canonical & Default Robots Settings
Hello! I'm working with Magento 1.9 for an eCommerce site with several hundred products. Currently I understand it is best practices to use the Canonical tag, however I also have my default robots set to "Index, Follow". Will this cause an issue having product pages set to index, follow but also having a canonical tag included? What are some best practices regarding Magento default robots & canonical tags? Any help is appreciated.
On-Page Optimization | | BretDarby0 -
Getting Duplicate meta title, and meta description errors
I have a website with shopify CMS. At the run time some pages generating itself with Tags name ( or we can say it search pages) which has been used in collections. so it's Meta tag and description is same as well as collection and in SEO tools it showing duplicate meta title, description errors. 1. http://brok3nclothing.com/collections/tanks/girls-skate-clothing
On-Page Optimization | | 1akal
2. http://brok3nclothing.com/collections/accessories/girls-skate-clothing
3. http://brok3nclothing.com/collections/accessories/girls-skate-clothing
4. http://brok3nclothing.com/collections/tanks/girls-surf-clothing For example above URLs are not on website, its generating by self with tags name like girls-skate-clothing, girls-surf-clothing etc. and these pages are accessing Meta titles, description from http://brok3nclothing.com/collections/tanks/ So will you please suggest me that should I worry about these duplicate meta title, description errors ???0 -
Meta description not showing in search results
Hi, One of my client's websites is not showing its Meta descriptions in search results. I've looked at the html and have noticed that they are using ' rather than " around the description tag (see below) <meta name="<a class="attribute-value">description</a>" content='<a class="attribute-value">Student Accommodation in Portsmouth or Southampton - Fully Furnished Studios, Close to the Universities. All Inclusive Rent with WiFi. Going Fast, Book Now!</a>' /> Is this likely to be the problem, or am I missing something very obvious? The website is http://www.unilife.co.uk/ Many thanks, Ben
On-Page Optimization | | bendyman0 -
Duplicate meta data for 301 redirected items
Google webmaster tools tells me that I have duplicate title tags for two items. They have the same product ID. When there is a change in SEO URL, the system does an automatic 301 redirect of the first item towards the second item, like in the example below:
On-Page Optimization | | Madlena
22"x28" White Foam Board School Project Kit|Geographics.com
/p48998/foam-board-school-project-kit-22-x-28/product_info.html /p48998/white-foam-board-school-project-kit/product_info.html What are we doing wrong that Google asks for different meta data for a 301 redirect? Thanks!0 -
Robots file include sitemap
Hello, I see that google, facebook and moz... have robots.txt include sitemap at the footer.
On-Page Optimization | | JohnHuynh
Eg: http://www.google.com.vn/robots.txt Sitemap: http://www.google.com/sitemaps_webmasters.xml
Sitemap: http://www.google.com/ventures/sitemap_ventures.xml Should I include my sitemap file (sitemap.xml) at the footer of robots.txt and why should do this? Thanks,0 -
Can you expound why i have to avoid using meta keywords?
I'm using the on page report card and it tells me that i have to avoid using meta keywords.I'm a little bit confused. I thought that it's important to use it all the time so search engine can better index the site. if I use SEO Quake it will tell me in the diagnostic test that I need to input keywords.
On-Page Optimization | | jsevilla0 -
Using AJAX to get a meta description to show up
We're unhappy with the meta descriptions google is picking up for our links in SERPs so have started using AJAX to stream in the content google was previously picking up for meta descriptions. This worked and it doesn't seem to have impacted traffic coming to our site, however since the day of that change our bounce rates have gone up significantly, even for pages that we did not push this change to. Is it possible that doing this caused Google to treat our site differently site wide? Is there anything we should be cautious of when doing this? I know the bounce rates could be impacted by users being better prepared by the google meta descriptions, however it doesn't explain what's happening to parts of our site that we didn't do anything to. -Billy
On-Page Optimization | | RealSelf0
