How can I prevent Google and other search engines to crawl my secured pages (https:)?
-
Let me know your thoughts guys. Thanks in advance!
-
Your best bet is to place a meta noindex tag on each secure page. If it's only a few pages, you could just add it by hand. If it's many, you should be able to access each pages protocol with whatever server-side language you're using, and dynamically add it on all secure pages.
If you use robots.txt to exclude the pages, Google can still show them in search results, with the description below them that reads, "A description for this result is not available because of this site's robots.txt – learn more." Personally, I don't care for that.
-
Hi there, blocking the HTTPS version of your pages from being crawled by the search engines is a bit tricky. You might need to come up with a separate robots.txt file to handle the HTTPS requests.
Here you go to know more about the process:
http://www.seoworkers.com/seo-articles-tutorials/robots-and-https.html
Hope it helps.
Best,
Devanur Rafi
-
Hi esiow
You have a choice of placing a robots.txt file in the root folder of your website or if blocking individual pages you could use the meta robots tag. See these page for more information: http://moz.com/learn/seo/robotstxt and https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag?csw=1
I hope that helps,
Peter
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEO Problem With My Pages ?
Hi everyone, I created a self-hosted custom PHP website the page is ( custom cosmetic boxes ), but it doesn't show up on Google. I tried to follow tutorials on the HubSpot, but even my site sucks. Please help me where I am wrong. Maybe the little thing is left unattended, which makes it void.
Web Design | | fbowable
Please help me.0 -
Are slides how's etc the new Splash Pages?
I just did SEO audits of approx 50 websites in the tourism sector. Nearly all had poor Google Pagespeed ratings, partly down to that, among other factors. I also feel that slideshows,, large images and videos in headers are poor for usability. I say get the content people need to engage with in front of them asap Are there any stats or studies that can provide insight on this? I've been telling those with these designs to keep an eye on bounce rates and let that guide them
Web Design | | anndonnelly0 -
We added hundreds of pages to our website & restructured the layout to include 3 additional locations within the sub-pages, same brand/domain name. How long could Google take to crawl/index the new pages and rank the keywords used within those pages?
We added hundreds of pages to our website & restructured the layout to include 3 additional locations within the sub-pages, same brand/domain name. The 3 locations old domains were redirected to their sites within our main brand domain. How long could Google take to crawl/index the new pages and rank the keywords used within those pages? And possibly increase our domain authority hopefully? We didn't want our brand spread out over multiple websites/domains on the internet. This also allowed for more content to be written on pages, per each of our locations service's, as well.
Web Design | | BurgSimpson0 -
Page missing meta description some times a good thing?
Just courious to why http://moz.com/learn/seo does not appear to have the meta tag: name="description" content=""> Is this purposely done this way for some reason? Or has it become less important? Just curious.
Web Design | | DanielMulderNL0 -
Wordpress: Pages vs Posts vs Portfolio
Hi All, I'm looking to put pen to paper and design my main structual template for my website. I will be creating the new site in Wordpress. My understanding of Wordpress is broken into the Static Pages, Posts and Portfolio. Static PAGE
Web Design | | Mark_Ch
Static one off content.
No tags, categories or archived Posts
content entries, which is listed in reverse chronological order.
Update post entry to maintain overall freshness of your website.
tags, categories & archived Portfolio
????? Question What are the benefits of a portfolio page over Static Pages & Posts When creating feature rich articles should i use static pages, posts or portfolio. Thanks Mark0 -
Pages vs. Posts for SEO
Hi, I would like your thoughts about pages vs. posts for SEO. I understand the difference in terms of WP structure and have read the SEOmoz blog post about setting up your site for SEO success (http://www.seomoz.org/blog/setup-wordpress-for-seo-success). However, if you're trying to rank for a particular keyword, it seems that either one could work, from an on-page SEO perspective, as far as title tag, URL, meta description, etc. So how do you decide whether to set up a page vs. a post? What are the pros and cons, from an SEO perspective, about using one vs. the other? Thanks in advance! Carolina
Web Design | | csmm0 -
Page Redirection solution
Page Redirection solution needs, there are 2 sites in the same folder and one page of old site is bxxxxxseo.com/products.php new site bxxxxxseo.com/product_list.php .there are many old page indexed i wanna redirect all old pages to relevant pages of new site using SEO friendly way .Any help really appropriate. Thank you
Web Design | | innofidelity0