Robots.txt on http vs. https
-
We recently changed our domain from http to https. When a user enters any URL on http, there is an global 301 redirect to the same page on https.
I cannot find instructions about what to do with robots.txt. Now that https is the canonical version, should I block the http-Version with robots.txt?
Strangely, I cannot find a single ressource about this...
-
Glad to be of help. Check out this Google link to confirm you picked up the 180 day crawl
https://support.google.com/webmasters/answer/83106?hl=en
Second URLs helpful as well.
http://blog.raventools.com/moving-site-from-http-to-ssl/
all the best,
tom
-
Good point with the backlinks! Currently, both robots.txt files are open and google does not seem to have canonicalization problems so far. So it makes sense to leave it this way anyways... Thanks Thomas!
-
"Now that https is the canonical version, should I block the http-Version with robots.txt?"
Absolutely not GWT will handel all of it think about backlinks both https:// & http:// urls you will not want to lose the flow of link juice that you would cut off
Remake robost.txt with
http://www.internetmarketingninjas.com/seo-tools/robots-txt-generator/
But use https:// for the xml sitemap.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robot.txt : How to block a specific file type in several subdirectories ?
Hello everyone ! I need help setting up a robot.txt. I'm trying to block all pdf files in particular directories so I'm using this command. In the example below the line is blocking all .gif in the entire site. Block files of a specific file type (for example, .gif) | Disallow: /*.gif$ 2 questions : Can I use this command to specify one particular directory in which I want to block pdf files ? Will this line be recognized by googlebots ? Disallow: /fileadmin/xxxxxxx/xxx/xxxxxxx/*.pdf$ Then I realized that I would have to write as many lines as many directories there are in which I want to block pdf files. Let's say I want to block pdf files in all these 3 directories /fileadmin/directory1 /fileadmin/directory1/sub1 /fileadmin/directory1/sub1/pdf Is there a pattern-matching rule I could use to blocks access to pdf files in all subdirectories instead of writing 3x the above line for each subdirectory ? For exemple : Disallow: /fileadmin/directory1*/ Many thanks in advance for any insight you may have.
Technical SEO | | LabeliumUSA0 -
From: http://www. to https://
Hi all, I am changing my hosting for legal and SEO reasons from http://www to https:// . Now I hear different stories on the redirects: 1: should i try and change my backlinks? 2: internally all links will be 301 redirected at first. Than I want to (manually) change them. It;s within Wordpress so there should be a plugin for this. Tips? 3: Will it affect my rankings and for what period? What I now know that at first it will drop little but eventually you will rank higher than before. Thanks so much in advance! Tymen
Technical SEO | | Tymen1 -
404 vs 410 vs 301
Hi guys, I am managing a real estate website, and obviously we have a LOT of pages detailing each property. As those properties get sold and removed from the website, I'm wondering how best to handle this - I know 404, 410 and 301's are all valid ways to go, but I want to provide the best UX combined with the best SEO effect. My thinking is to customise a 410 page to show the page has been permanently removed, and has a relevant message (rather than a generic 404 message) and shows a search box - possibly pre-populated according to the page they were looking for.
Technical SEO | | LoonyToons
I think this gives a good UX and helps Google to understand the importance of the 000's of pages on our website.
I'd also like to clear property detail 404's as quick as possible to make it easier to see if we have problems elsewhere on the site. Having explained this to our development/SEO agency, they are strongly pushing for 301 redirects or leave as 404.
I think 301's would be the worst for UX, and as explained earlier, the volume of 404's is massive and makes it difficult to see real errors. They seem to think this is a better UX and better for SEO. Just wondering what you guys would recommend?0 -
SEO best practice : HTTP to HTTPS
What's the best practice to switch from an all HTTP site to an all HTTPS site ?
Technical SEO | | Crocodesign
No changes to the site structure, just a full site switch to SSL.
Right now, the site is reachable with HTTP and with HTTPS. http://crocodesign.be --> https://crocodesign.be
http://www.crocodesign.be --> https://crocodesign.be
https://www.crocodesign.be --> https://crocodesign.be CMS : Wordpress 3.9
Server type : Apache
Preferred method : .htaccess0 -
The use of robots.txt
Could someone please confirm that if I do not want to block any pages from my URL, then I do not need a robots.txt file on my site? Thanks
Technical SEO | | ICON_Malta0 -
Which is better Title length vs. keywords?
We run a jobboard. The title tag on a page for a job is often over 70 characters. An example of one would be: " Supplier Quality Inspector (Electrical Manufacturing) Job in Orlando, FL 32809 at Pro Image Solutions | Orlando Jobs!" The company name 'Orlando Jobs!" comes at the end but is also a really good keyword e.g. 'Orlando' and 'Jobs' I am interested in suggestions as to how to make these titles better. For example take off the company name when we go over 70 characters? Move the company name to the front of the title because the company name is also good keywords? I am looking for the best way to handle the issue is all. Thanks.
Technical SEO | | JobBiz0 -
SEO Terms for Internal Vs External
Hey there! I am writing up an SEO plan for our company and wanted to get the groups input on the use of some SEO terms. I need to organize and explain these efforts to nonSEO people. I usually talk about, SEO in terms of "Internal" vs "External" efforts. Internal SEO efforts being things like Title Tags, Description Tags, Page Speed, Minimizing errors, proper 301 redirect, content development for the site, internal linking and anchor, etc. External SEO efforts being things like Link building, social media profile setups and posts (FB Twitter Pinterest, YouTube), PR work. How do you split these out? What terms do you use? Do you subdivide these tasks? What terms do you use? For example, with Internal, I sometimes talk about "Technical SEO" that has do to with making sure that site speed is working well, 301s are setup correctly, noindex tag etc are all used properly. These are things that different versus "On Page" efforts to use keywords properly etc. I will also use the term "Site Visibility" for non SEOs to explain the technical impact. For example, if your site has the wrong robots.txt, if you have 500 errors everywhere and a slow site, if you are sending spiders down a daisy chain of 301s, it is difficult for the key parts of your site to be found and so your "Visibility" to the engines are poor. You have to get your visibility up, before you begin to then worry about if you have the right keywords on a page etc. Any input or references would be appreciated.
Technical SEO | | CleverPhD0 -
HTTPS attaching to home page
Hi!! Okay - weird tech question. Domain is http://hiphound.com. I have SSL attaching to checkout and my account pages. Tested and works well. Issue - I am able to reach the home page at https://hiphound.com AND http://hiphound.com. If I access the home page via HTTPS and click on a link (any link) then the site is redirected to HTTP again which is good. My concern is the home page displaying via HTTPS and HTTP. Is this is an issue that can be resolved or is it expected behavior I have to live with.? I am being told by DEV there is nothing they can do about it but want to understand why and if they are correct. Thoughts? Thank you!! Lynn
Technical SEO | | hiphound0