How can I prevent Google and other search engines to crawl my secured pages (https:)?
-
Let me know your thoughts guys. Thanks in advance!
-
Your best bet is to place a meta noindex tag on each secure page. If it's only a few pages, you could just add it by hand. If it's many, you should be able to access each pages protocol with whatever server-side language you're using, and dynamically add it on all secure pages.
If you use robots.txt to exclude the pages, Google can still show them in search results, with the description below them that reads, "A description for this result is not available because of this site's robots.txt – learn more." Personally, I don't care for that.
-
Hi there, blocking the HTTPS version of your pages from being crawled by the search engines is a bit tricky. You might need to come up with a separate robots.txt file to handle the HTTPS requests.
Here you go to know more about the process:
http://www.seoworkers.com/seo-articles-tutorials/robots-and-https.html
Hope it helps.
Best,
Devanur Rafi
-
Hi esiow
You have a choice of placing a robots.txt file in the root folder of your website or if blocking individual pages you could use the meta robots tag. See these page for more information: http://moz.com/learn/seo/robotstxt and https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag?csw=1
I hope that helps,
Peter
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are slides how's etc the new Splash Pages?
[How did Moz know that my question was about this?!] I've just completed an audit of nearly 50 websites in the tourism industry and 90% had a slideshow, large image or video taking up more than the initial screen on the fairly large screened Chromebook that I'm using. I'm advising them all to ditch this and am often getting resistance from the site owners and their web developers. I know that these can be better optimized for page load speed, which is poor for most of these sites, especially on mobile devices; but from a usability standpoint, are these affective at drawing in users? Do users take the time to view these? Are they annoyed at always having to scroll down to see if there is anything else useful on the homepage? I think they are like the splash pages of the past: poor for usability and SEO. I've advised to at least make sure that the images are sized so the top of the page fits any screen (some of them do resize well for mobile devices, but maybe not laptops/desktops), include text with calls to action and click through to relevant content. I've been noting that they aren't media businesses selling images or videos, so they need to get their offerings to the top of the page so that users can see and engage more quickly. Anyone have any stats or experience on this? Thanks, Ann
Web Design | | anndonnelly0 -
I am Using <noscript>in All Webpage and google not Crawl my site automatically any solution</noscript>
| |
Web Design | | ahtisham2018
| | <noscript></span></td> </tr> <tr> <td class="line-number"> </td> <td class="line-content"><meta http-equiv="refresh" content="0;url=errorPages/content-blocked.jsp?reason=js"></td> </tr> <tr> <td class="line-number"> </td> <td class="line-content"><span class="html-tag"></noscript> | and Please tell me effect on seo or not1 -
Doing SEO for single page applications / Prerender.io
My dev and I are migrating an existing multi page application to a single page application with prerender.io. Does anybody have any experience with doing SEO for single page applications? Any other consequences we should take into account? Anything important to expect. Any insights would be 10/10 appreciated.
Web Design | | Edward_Sturm0 -
Can forwarding users from one domain to a different domain damage rank and authority of first domain?
Preliminary Explanation: We launched a new website a couple months back but haven't had much luck in Google taking notice. One of the main attractions to our site is an old flash app that was made nearly a decade ago. As the original developer has long ago moved on and we are unable to figure out how to integrate it with our new site, we've been stuck hosting the flash app on a different domain. As such, users who come to our site and want to use the app must immediately navigate away from our site to this other domain. This has caused our primary domain's bounce rate and average site time to plummet while raising it for the other domain. My question: is this damaging our search rank and page authority with Google for this primary domain/site and counter-acting any other positive SEO changes we can make? How much weight does Google give towards bounce rate/average site time spent by users in its overall calculations for search rank and page authority? Our average site time for this primary domain is resting currently at 50-60 seconds, while for the secondary domain that hosts the old flash app it is 4-5 minutes.
Web Design | | Closetstogo0 -
No cache meta tags - does it help Google get back and reindex faster?
I saw these meta tags on a site and am trying to figure out their benefit. These meta tags are on the home page, product pages, every page of the site. Will it cause search engine bots to come back and index pages faster? Will it cause slower page loading in browsers if nothing is cached? http-equiv="pragma" content="no-cache"/> http-equiv="cache-control" content="no-cache,no-store,must-revalidate"/> http-equiv="expires" content="0"/>
Web Design | | CFSSEO0 -
Funnel tracking with one page check-out?
Hi Guys, I'm creating a new website with a one page checkout that follows the following steps:
Web Design | | Jerune
1. Check availability
2. Select product
2. Select additional product & Add features
3. Provide personal information
4. Order & Pay I'm researching if it is possible to track all these steps (and even steps within the steps) with Google Analytics in order to analyse checkout abandonment. The problem is only that my one-page checkout has only one URL (I want to keep it that way) and therefore can not be differentiated on URL in the Analytics funnel. To continue to the next step also the same button (in a floating cart) in used to advance. The buttons to select/choose something within one step are all different. Do you guys know how I can set this up and how detailed I can make this? For example, is it also possible to test at which field visitors leave when for example filling in their personal information? Would be great if you can help me out!0 -
Can Image File Names be Masked?
If we "mask" file names for our website but they are left their original name on the server, will Google notice this? Client wants to mask them in order to name them with keywords but not change on the actual server.
Web Design | | Atlanta-SMO0 -
Writing A Data Extraction To Web Page Program
In my area, there are few different law enforcement agencies that post real time data on car accidents. One is http://www.flhsmv.gov/fhp/traffic/crs_h501.htm. They post the accidents by county, and then in the location heading, they add the intersection and the city. For most of these counties and cities, our website, http://www.kempruge.com/personal-injury/auto-and-car-accidents/ has city and county specific pages. I need to figure out a way to pull the information from the FHP site and other real time crash sites so that it will automatically post on our pages. For example, if there's an accident in Hillsborough County on I-275 in Tampa, I'd like to have that immediately post on our "Hillsborough county car accident attorney" page and our "Tampa car accident attorney" page. I want our pages to have something comparable to a stock ticker widget, but for car accidents specific to each pages location AND combines all the info from the various law enforcement agencies. Any thoughts on how to go about creating this? As always, thank you all for taking time out of your work to assist me with whatever information or ideas you have. I really appreciate it.
Web Design | | KempRugeLawGroup0