How can I prevent Google and other search engines to crawl my secured pages (https:)?
-
Let me know your thoughts guys. Thanks in advance!
-
Your best bet is to place a meta noindex tag on each secure page. If it's only a few pages, you could just add it by hand. If it's many, you should be able to access each pages protocol with whatever server-side language you're using, and dynamically add it on all secure pages.
If you use robots.txt to exclude the pages, Google can still show them in search results, with the description below them that reads, "A description for this result is not available because of this site's robots.txt – learn more." Personally, I don't care for that.
-
Hi there, blocking the HTTPS version of your pages from being crawled by the search engines is a bit tricky. You might need to come up with a separate robots.txt file to handle the HTTPS requests.
Here you go to know more about the process:
http://www.seoworkers.com/seo-articles-tutorials/robots-and-https.html
Hope it helps.
Best,
Devanur Rafi
-
Hi esiow
You have a choice of placing a robots.txt file in the root folder of your website or if blocking individual pages you could use the meta robots tag. See these page for more information: http://moz.com/learn/seo/robotstxt and https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag?csw=1
I hope that helps,
Peter
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are slides how's etc the new Splash Pages?
I just did SEO audits of approx 50 websites in the tourism sector. Nearly all had poor Google Pagespeed ratings, partly down to that, among other factors. I also feel that slideshows,, large images and videos in headers are poor for usability. I say get the content people need to engage with in front of them asap Are there any stats or studies that can provide insight on this? I've been telling those with these designs to keep an eye on bounce rates and let that guide them
Web Design | | anndonnelly0 -
Will Google Judge Duplicate Content on Responsive Pages to be Keyword Spamming?
I have a website for my small business, and hope to improve the search results position for 5 landing pages. I recently modified my website to make it responsive (mobile friendly). I was not able to use Bootstrap; the layout of the pages is a bit unusual and doesn't lend itself to the options Bootstrap provides. Each landing page has 3 main div's - one for desktop, one for tablet, one for phone.
Web Design | | CurtisB
The text content displayed in each div is the same. Only one of the 3 div’s is visible; the user’s screen width determines which div is visible. When I wrote the HTML for the page, I didn't want each div to have identical text. I worried that
when Google indexed the page it would see the same text 3 times, and would conclude that keyword spamming was occurring. So I put the text in just one div. And when the page loads jQuery copies the text from the first div to the other two div's. But now I've learned that when Google indexes a page it looks at both the page that is served AND the page that is rendered. And in my case the page that is rendered - after it loads and the jQuery code is executed – contains duplicate text content in three div's. So perhaps my approach - having the served page contain just one div with text content – fails to help, because Google examines the rendered page, which has duplicate text content in three div's. Here is the layout of one landing page, as served by the server. 1000 words of text goes here. No text. jQuery will copy the text from div id="desktop" into here. No text. jQuery will copy the text from div id="desktop" into here. ===================================================================================== My question is: Will Google conclude that keyword spamming is occurring because of the duplicate content the rendered page contains, or will it realize that only one of the div's is visible at a time, and the duplicate content is there only to achieve a responsive design? Thank you!0 -
Address On Every page for e-Commerce site?
For a primarily e-commerce site, should you have your address on every page (in the footer, for example)? Or is it enough to just have it on the contact page? Thanks, Ruben
Web Design | | KempRugeLawGroup0 -
Too Many Links on One Page - What to Do?!
Hello Geniuses, Prodigies, and Experts of the Field, My website pages for www.1099pro.com have too many links on one page, something like 150-175, and I understand that each page should ideally be under 100. Most of these links, approx 105, come from dropdown navigation options in the header toolbar or the footer links. It is my take that these links make our site easier to navigate but I'm sure that they are hurting my pagerank / SERPs. Is there a best way to handle a situation like this? I'd really prefer not to alter the header/footer layout of the entire site by removing 50-75 navigational links. The only other idea I have is below but I have no idea if it would work. For any link that I do not care to pass pagerank, institute a "nofollow" parameter. This would be my favorite option if it is viable.
Web Design | | Stew2220 -
Accordion Fold Ups Bad For Google
http://fandicoach.com/products Right now I have these accordion things on the website. Are they bad for google in terms of being an SEO best practice? I want to avoid doing anything black hat. Thanks!
Web Design | | OOMDODigital0 -
Do pages with low PA effect DA?
I was looking into raising my firm's Domain Authority and I had a thought. I was wondering if our very low PA pages are bringing the overall DA down? Our homepage is at 43 currently and the DA is 32. We have quite a few pages and I am trimming a little fat in the deep pages currently. I was wondering what would be the result in terms of DA I am also starting an aggressive initiative to blog more and try to attract links through guest posting and HARO (Hep A Reporter Out). I understand that many people will say DA is not a metric your should necessarily build around. But, while I am fighting for rankings in a very competitive vertical, I assume a higher DA is better, no? From everything I have read on Moz over the years, they say that the DA metric is the one that tries to encompass a multitude of factors, similar to the way something more complex like the Google algorithm does. So, I assumed finding small gains in DA could be beneficial to the site's rankings and traffic I tried to go into detail and get specific here because I know how many bad questions are asked daily. Thanks anyone and everyone for the help, I do thoroughly appreciate the Moz community
Web Design | | BossArrighi0 -
Websites with only one "html file" and page href # is good for SEO?
I bought one website from templatemonster that contains only one HTML and the pages are generated by links (PROGRAMACAO) My website: www.nextformaturas.com.br This is good in term of SEO? or it is better an website with deveral pages with diferent contents? What are the pros and cons? I really lost on this.
Web Design | | Naghirniac0