SSL and robots.txt question - confused by Google guidelines
-
I noticed "Don’t block your HTTPS site from crawling using robots.txt" here: http://googlewebmastercentral.blogspot.co.uk/2014/08/https-as-ranking-signal.html
Does this mean you can't use robots.txt anywhere on the site - even parts of a site you want to noindex, for example?
-
Hi Luke,
Just make sure that your robots.txt file located at https://www.example.com/robots.txt doesn't block search engine spiders. Of course there may be some folders or filetypes you want to block but it certainly shouldn't look like below which would block everything:
User-agent: *
Disallow: /
Hope that helps
-
No that's not what they mean - it means Google recommends you allow the secure version of your site(where applicable) to be crawled. You can still block certain pages/sections should you choose to do so.
With regards to noindexing you could also place this on the actual page as an alternative.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
URL Structure Question
Am starting to work with a new site that has a domain name contrived to help it with a certain kind of long tail search. Just for fictional example sake, let's call it WhatAreTheBestRestaurantsIn.com. The idea is that people might do searches for "what are the best restaurants in seattle" and over time they would make some organic search progress. Again, fictional top level domain example, but the real thing is just like that and designed to be cities in all states. Here's the question, if you were targeting searches like the above and had that domain to work with, would you go with... whatarethebestrestaurantsin.com/seattle-washington whatarethebestrestaurantsin.com/washington/seattle whatarethebestrestaurantsin.com/wa/seattle whatarethebestrestaurantsin.com/what-are-the-best-restaurants-in-seattle-wa ... or what and why? Separate question (still need the above answered), would you rather go with a super short (4 letter), but meaningless domain name, and stick the longtail part after that? I doubt I can win the argument the new domain name, so still need the first question answered. The good news is it's pretty good content. Thanks... Darcy
Intermediate & Advanced SEO | | 945010 -
Question about using abbreviation
Hello, I have this abbreviation inside my domain name, ok? now for a page URL name, do you recommend me to use the actual word (which shortened form of it is inside domain name) in a page name? Or when have abbreviation in domain name, then using its actual word in a page name is not good? It's all about how much google recognize abbreviation as the actual word and gives the same value of word to it? do I risk not using the actual word? Hope made myself clear ) thanks.
Intermediate & Advanced SEO | | mdmoz0 -
Google showing high volume of URLs blocked by robots.txt in in index-should we be concerned?
if we search site:domain.com vs www.domain.com, We see: 130,000 vs 15,000 results. When reviewing the site:domain.com results, we're finding that the majority of the URLs showing are blocked by robots.txt. They are subdomains that we use as production environments (and contain similar content as the rest of our site). And, we also find the message "In order to show you the most relevant results, we have omitted some entries very similar to the 541 already displayed." SEER Interactive mentions that this is one way to gauge a Panda penalty: http://www.seerinteractive.com/blog/100-panda-recovery-what-we-learned-to-identify-issues-get-your-traffic-back We were hit by Panda some time back--is this an issue we should address? Should we unblock the subdomains and add noindex, follow?
Intermediate & Advanced SEO | | nicole.healthline0 -
Another footer question
Hi to all, Maybe this question is already answered (in that case sorry) but I didn't find it. Currently, with the latest changes is really useful to have a 'seo footer'. I mean, it seems that can give you more problems that benefits. In my case the idea of the footer is only to obtain more traffic. Having this in mind, I'm right thinking that is better don't write anything ? Thanks in advance
Intermediate & Advanced SEO | | nsimpson0 -
Rank on specific Google
Hi folks, a website is hosted with a TLD like .com in the USA. The content etc. is obviously all english but now we want to focus on a specific Google like .co.uk What must be necessarily be done to rank better? Is it enough just to buy a .co.uk domain and set the nameserver up or do we need to get a british hosting? Thanks in advance. Mike
Intermediate & Advanced SEO | | KillAccountPlease0 -
A few questions regarding listings in Google Places
For an SAB (Service Area Business) with a hidden address - Can you have more then one listing? Can you use a free Google Voice number? Can you forward the number to a main number? Can the listing be in an office building? Such as a rented space... For a non SAB listing with the address visible - Can you use free Google voice numbers for each listing and forward them to one main number?
Intermediate & Advanced SEO | | Bryan_Loconto0 -
Question about copying content
Hi there, I have had a question from a retailer asking if they can take all our content i.e. blog articles, product pages etc, what is best practice here in getting SEO value out of this? Here a few ideas I was thinking of: I was thinking they put canonical tags on all pages where they have copied our content? They copy the content but leave all anchor text in place? Please let me know your thoughts. Kind Regards
Intermediate & Advanced SEO | | Paul780 -
Is Google mad at me for redirecting...?
Hi, I have an e-commerce website that sells unique items (one of a kind). We have hundreds of items and the items are rapidly sold. Up till now I kept the sold items under our "sold items" section but it started to get back at me as we have more "sold" than non sold and we are having duplication problems (the items are quite similar besides to sizes etc.). What should we do? Should we redirect 100 pages each week? Will Google be upset with that? (for driving it crazy) Thanks
Intermediate & Advanced SEO | | BeytzNet0