Blocked By Meta Robots
-
Hi
I logged in the other day to find that over night I received 8347 notices saying certain pages are being kept out of the search engine indexes by meta-robots.
I have not changed my robots.txt in years and I certainly didn't block Google from visiting those pages.
Is this a fault on Roger Mozbot behalf? Or is there a bot preventing 8000+ pages being indexed?
Is there a way to find out what meta-robot is doing this and where? And how I can get rid of it?
I usually rank between #3 and #5 for the term 'sex toys' on google.com.au, but I now rank #7 to #9 so it would seem some of my pages/content is being blocked.
My website is www.theloveshop.com THIS IS AN ADULT TOYS SITE.
There is no porn videos or anything like that on it, but just in case you don't wish to look at sex toys or are around kids I thought I would mention it.
Blake
-
Hi Nakul
Thank you very much for the suggestions and I will start checking the code on those pages today. Thanks for heads up on the .com.au site as well I will be changing the hosting today to Aus, and thanks to you that site may get pushed up the rankings for a highly competitive term that I have been fighting to get for years
As for following my competitors links, I have been doing that for awhile and it would seem most of them either buy their links or have them on pages that are irrelevant to our niche, one of my competitors has even acquired more links than SEOmoz.com but has no were near the content. In saying that I am by no means a pure white hat SEO, more of a grey hat if such a thing exists, simply because I find it hard to get links to an adult site without it being spammy of some sorts. So I have started writing articles and linking to products mentioned, but trying to make the articles interesting/funny at the same time. Hopefully soon I will start to see some improvement.
Again thank you for your help and I will let you know how I go with those pages blocked by bots.
Blake
-
The meta robots block sounds like it's a block from your CMS or something in your header include file or something. I would check on the pages that show as blocked to see if there's any noindex sort of code in those pages. Other then that, shift in rankings could be based on your competition optimizing for the same set of keywords. I would recommend to spend some time analyzing the backlinks of the competition and see if you can gain some of those kinds of links.
On a side note: It also looks like you own both the .com and the .com.au site. The .com is hosted in AU and the .com.au is hosted in USA. Did you intend to do it the other way around ? I would imagine your rankings for the .com.au get better if it was hosted in Australia instead of USA.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Nofollow Meta Tag
Hello, I'm new to this forum so please forgive if this has been discussed before. I have a question about the nofollow meta tag being used at the page level. We have external links within our website's guest blog pages that we do not want followed. I would use a nofollow rel attribute on the link itself, but since we allow the blog to be posted using a wysiwyg-type text editor, that proves to be a little difficult to enforce. I was thinking about using the nofollow meta tag at the page level to handle this, but was a little unsure on how that may affect our own links...such as menu and side-bar links...that are on those pages. I'm not too concerned about those internal links not being followed on those particular blog pages, but I wanted to make sure a nofollow for a link on one page won't have any negative side effects on those same menu links that appear on various other pages without the nofollow meta tag. Would there be any negative side effects to using the nofollow meta tag like that? Thanks in advance for any insight. Best Regards, Ken
On-Page Optimization | | kens1090 -
Duplicate 'meta title' issue (AMP & NON-AMP Pages)
how to fix duplicate meta title issue in amp and non-amp pages? example.com
On-Page Optimization | | 21centuryweb
example.com/amp We have set the 'meta title' in desktop version & we don't want to change the title for AMP page as we have more than 10K pages on the website. ----As per SEMRUSH Tool---- ABOUT THIS ISSUE It is a bad idea to duplicate your title tag content in your first-level header. If your page’s <title>and <h1> tags match, the latter may appear over-optimized to search engines. Also, using the same content in titles and headers means a lost opportunity to incorporate other relevant keywords for your page.</p> <p><strong>HOW TO FIX IT</strong></p> <p>Try to create different content for your <title> and <h1> tags.<br /><br />this is what they are recommending, for the above issue we have asked our team to create unique meta and post title for desktop version but what about AMP page?<br /><br />Please help!</p></title>0 -
Meta tag keywords with the same words in them.
I'm updating some older pages and was wondering about potential penalties from having keywords that start with the same phrase. It's a geographic area so there is the "full name" and the abbreviated name. I'd like to have keywords for both. For example: virginia beach, va beach, virginia beach attraction, virginia beach things to do, va beach attraction, va beach things to do, virginia beach dolphins tour, va beach dolphins tour Is that spammy? I understand they don't have the same weight as they used to but I'd like to optimize for them anyway since I'm redoing some things. Thanks in advance.
On-Page Optimization | | recoil0 -
Login webpage blocked by robots
Hi, the SEOMOZ crawl diagnostics shows that this page: www.tarifakitesurfcamp.com/wp-login.php is blocked (noindex, nofollow) Is there any problem with that?
On-Page Optimization | | juanmiguelcr0 -
Downloading Meta Data
I'm about to do a site audit, and as part of it I need to download all the metadata so that I can see what is ok, and what needs to change. I've seen tools that allow you to do this a million times, and even used a few. But today for some reason I'm drawing a blank. Can someone help me please? Best case scenario, the tool is free, but I'm not opposed to paying if it's going to save me a lot of time.
On-Page Optimization | | ZephSnapp0 -
Meta Data definition for multiple pages. Potential duplicate content risk?
Hi all, One of our clients needs to redefine their meta title and description tags. They publish very similar information almost every day, so the structure they propose is the following: Structure 1: Type of Analysis + periodicity + data + brand name Examples 1: Monthly Market Analysis, 1/5/2012 - Brand Name Weekly Technical Analysis, 7/5/2012 - Brand Name Structure 2: Company Name + investment recommendation + periodicity Example 2: Iberdrola + investment recommendation (this text doesn't vary) + 2T12 (wich means 2012, 2nd trimestrer) Regarding meta description they want to follow a similar approach, replicating every time the same info with a slight variation for each publication. I'm afraid this may cause a duplicate content problem because of the resemblance of every "Market Analysis" done or every "Investment recommendation" done in the future. My initial suggestion for them is to define specific and unique meta data for each page, but this is not possible for them given the time it takes to do it for every page. Finally, I ask them to specify the data in each meta title of content published, in order to add something different each time and avoid duplicate content penalty. Will this be enough to avoid duplicate content issues? Thanks in advance for your help folks! Alex
On-Page Optimization | | elisainteractive0 -
Meta Descriptions - Duplicate Content?
I have created a Meta Description for a page that is optimized for SERPS. If I also put this exact content on my page for my readers, would this be considered duplicate content? The meta description and content will be listed on the same page with the same URL. Thanks for your help.
On-Page Optimization | | tuckjames0 -
Are Amazon meta tags efficient?
We are probably all familiar with general and Google guidelines for writing title and description tags. But Amazon. com often create another structure where they put in a) amazon.com, b) product name or description and c) the Amazon category the product is featured in, like this: | Amazon.com: Mac Motion Chairs Model 2-Piece Recliner with Matching Ottoman Mocha Microfiber with Walnut Frame: Home & Garden Is this a well developed description tag? |
On-Page Optimization | | KnutDSvendsen
|0