Disallowing https URLs
-
It there a problem disallowing all https URLs to be indexed in order to avoid duplication?
This is the article recommending this practice -
http://blog.leonardchallis.com/seo/serve-a-different-robots-txt-for-https/
Thanks!
-
no there is nothing wrong at all. Generaly you would either force pages to be either one or the other. There is not too many senarios where you would want to allow both.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Transfering Site from Http to HTTPS
Migrating all of our pages from HTTP to HTTPS. I am listing few of my concerns regarding the same: Currently, all HTTPS traffic to our Homepage and SEO page is 301 Redirected to HTTP equivalent. So, when we enable HTTPS on all our pages and 301 all HTTP traffic to HTTPS and stop current 301 Redirection to HTTP, will it still cause a loop during Google crawl due to old indexing? Will we move whole SEO facing site to HTTPS at once or will it be in phases? Which of the two approach is better keeping SEO in mind? what all SEO changes will be required on all pages.(eg. Canonical URLs on our website as well as affiliate websites), sitemaps etc.
Technical SEO | | RobinJA1 -
Should I disallow crawl of my Job board?
MOZ crawler is telling me we have loads of duplicate content issues. We use a Job Board plugin on our Wordpress site and we have allot of duplicate or very similar jobs (usually just a different location), but the plugin doesn't allow us to add any rel canonical tags to the individual jobs. Should I disallow the /jobs/ url in the robots.txt file? This will solve the duplicate content issue but then Google wont be able to crawl any of the individual job listings Has anyone had any experience working with a job board plugin on Wordpress and had a similar issue, or can advise on how best to solve our duplicate content?? Thanks 🙂
Technical SEO | | O2C0 -
URL path randomly changing
Hi eveyone, got a quick question about URL structures: I'm currently working in ecommerce with a site that has hundreds of products that can be accessed through different URL paths: 1)www.domain.com/productx 2)www.domain.com/category/productx 3)www.domain.com/category/subcategory/productx 4)www.domain.com/bestsellers/productx 5)... In order to get rid of dublicate content issues, the canoncial tag has been installed on all the pages required. The problem I'm witnessing now is the following: If a visitor comes to the site and navigates to the product through example 2) at time the URL shown in the URL browser box is example 4), sometimes example 1) or whatever. So it is constantly changing. Does anyone know, why this happens and if it has any impact on GA tracking or even on SEO peformance. Any reply is much appreciated Thanks you
Technical SEO | | ennovators0 -
Need URL structure suggestions
On my website I am in the process of creating expat city guides for different cities in Cambodia. I've already gotten three up, but I am worried that my URL structure is not the best, so I am wondering if I should fix it before I put the rest up. Right now the city guides are housed here: movetocambodia.com/expat-city-and-island-guides/ There's a section for each city, this one is for Battambang: movetocambodia.com/expat-city-and-island-guides/battambang And then there are sections for hotels, restaurants, etc. movetocambodia.com/expat-city-and-island-guides/battambang/battambang-hotels-and-accommodation So once you finally get to a review for an individual hotel or activity, the URL is really long, like this: movetocambodia.com/expat-city-and-island-guides/battambang/battambang-hotels-and-accommodation/classy-hotel Should I just par the section names down so the URL would be something like this: movetocambodia.com/expat-city-guides/battambang/accommodation/classy-hotel/ ? I was hoping by having the long URLs slugs for my section pages, such as "battambang-hotels-and-accommodation" they would be more likely to show on search terms like "Battambang hotels" than if the section was just "accommodation." However, this whole section is getting much less search traffic than anything else on my site, so I am wondering if it is because of these ridiculously long URLs. Any suggestions would be appreciated.
Technical SEO | | Lina5000 -
Redirect URLS with 301 twice
Hello, I had asked my client to ask her web developer to move to a more simplified URL structure. There was a folder called "home" after the root which served no purpose. I asked for the URLs to be redirected using 301 to the new URLs which did not have this structure. However, the web developer didn't agree and decided to just rename the "home" folder "p". I don't know why he did this. We argued the case and he then created the URL structure we wanted. Initially he had 301 redirected the old URLS (the one with "Home") to his new version (the one with the "p"). When we asked for the more simplified URL after arguing, he just redirected all the "p" URLS to the PAGE NOT FOUND. However, remember, all the original URLs are now being redirected to the PAGE NOT FOUND as a result. The problems I see are these unless he redirects again: The new simplified URLS have to start from scratch to rank 2)We have duplicated content - two URLs with the same content Customers clicking products in the SERPs will currently find that they are being redirect to the 404 page. I understand that redirection has to occur but my questions are these: Is it ok to redirect twice with 301 - so old URL to the "p" version then to final simplified version. Will link juice be lost doing this twice? If he redirects from the original URLS to the final version missing out the "p" version, what should happen to the "p" version - they are currently indexed. Any help would be appreciated. Thanks
Technical SEO | | AL123al0 -
Special characters in URL
Will registered trademark symbol within a URL be bad? I know some special characters are unsafe (#, >, etc.) but can not find anything that mentions registered trademark. Thanks!
Technical SEO | | bonnierSEO0 -
Https redirect when certificate expired
Hi, How do we 301 an https version of a domain to a page on another website when the security certificate has run out? We have 301 redirected the http version but IT stuck on how to do the expired https. Thanks
Technical SEO | | Houses0 -
URL parameter reduction plug in
Anyone know of a good plug-in that reduces the amount parameters used in URLs? I need one for an ASP based system and a PHP based system
Technical SEO | | matmox0