Development/Test Ecommerce Website Mistakenly Indexed
-
My question is - relatively speaking, how damaging to SEO is it to have BOTH your development/testing site and your live version indexed/crawled by Google and appearing in the SERPs?
We just launched about a month ago, and made a change to the robots text on the development site without noticing ... which lead to it being indexed too.So now the ecommerce website is duplicated in Google ... each under different URLs of course (and on diff servers, DNS etc)
We'll fix it right away ... and block crawlers to the development site. But again, may general question is what is the general damage to SEO ... if any ... created by this kind of mistake. My feeling is nothing significant
-
No my friend, no! I'm saying we'll point the existing staging/testing environment to the production version and will stop using it as staging instead of closing it completely like I mentioned earlier. And, we'll launch a fresh instance for staging/testing use case.
This will help us transferring majority if the link juice of already indexed staging/testing instance.
-
Why would you want to 301 a staging/dev environment to a production site? Unless you plan on making live changes to the production server (not safe), you'd want to keep them separate. Especially for eCommerce it would be important to have different environments to test and QA before pushing a change live. Making any change that impacts a number of pages could damage your ability to generate revenue from the site. You don't take down the development/testing site, because that's your safe environment to test changes before pushing updates to production.
I'm not sure I follow your recommendation. Am I missing a critical point?
-
Hi Eric,
Well, that's a valid point that bots might have considered your staging instances as the main website and hence, this could end up giving you nothing but a face palm.
The solution you suggested is similar to the one I suggested where we are not getting any benefit from the existing instance by removing it or putting noindex everywhere.
My bad! I assumed your staging/testing instance(s) got indexed recently only and are not very powerful from domain & page authority perspective. In fact, being a developer, I should have considered the worst case only
Thanks for pointing out the worst case Eric i.e when your staging/testing instances are decently old and you don't want to loose their SEO values while fixing this issue. And, here'e my proposed solution for it: don't removed the instance, don't even put a noindex everywhere. The better solution would be establishing a 301 redirect bridge from your staging/testing instance to your original website. In this case, ~90% of the link juice that your staging/testing instances have earned, will get passed. Make sure each and every URL of the staging/testing instance is properly 301 redirecting to the original instance.
Hope this helps!
-
It could hurt you in the long run (Google may decide the dev site is more relevant than your live site), but this is an easy fix. No-index your dev site. Just slap a site-wide noindex meta tag across all the pages, and when you're ready to move that code to the production site you remove that instance of code.
Disallowing from the robots.txt file will help, but that's a soft request. The best way to keep the dev site from being indexed is to use the noindex tag. Since it seems like you want to QA in a live environment that would prevent search engines from indexing the site, and still allow you to test in a production-like scenario.
-
Hey,
I recently faced the same issue when the staging instances got indexed accidentally and we were open for the duplicate content penalty (well, that's not cool). After a decent bit of research, I followed the following steps and got rid of this issue:
- I removed my staging instances i.e staging1.mysite.com, staging2.mysite.com and so on. Removing such instances helps you deindex already indexed pages faster than just blocking the whole website from robots.txt
- Relaunched the staging instances with a slightly different name like new-staging1.mysite.com, new-staging2.mysite.com and disallow bots on these instances from the day zero to avoid this mess again.
This helped me fixing this issue asap. Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Specific page does not index
Hi, First question: Working on the indexation of all pages for a specific client, there's one page that refuses to index. Google Search console says there's a robots.txt file, but I can't seem to find any tracks of that in the backend, nor in the code itself. Could someone reach out to me and tell me why this is happening? The page: https://www.brody.be/nl/assistentiewoningen/ Second question: Google is showing another meta description than the one our client gave in in Yoast Premium snippet. Could it be there's another plugin overwriting this description? Or do we have to wait for it to change after a specific period of time? Hope you guys can help
Intermediate & Advanced SEO | | conversal0 -
Website Redirection Issue
Hi All, Like to know is there any better way to do 301 redirection. My Client whose website name is Online Plants created with OpenCart. Over the period of time he added nearly 10,000's of products and now he is cleaning them ( by grouping similar attribute under one products) which is right way to do. For example , Product A with different size ( X,XL,XXL ) previously had 3 product entry ( A - X, A - XL, A - XXL ) , now he is moving all of them under one. So while moving he is deleting the other two entry. Now whats the best way to inform google . Putting a manual 301 redirection for each and every product is impossible as there are more products. Whats the best way to go ahead on this.
Intermediate & Advanced SEO | | Verve-Innovation1 -
How do I handle this 301/indexing mess?
I'm working on a client's site and noticed a brisk drop in rankings. In doing some digging I found that the homepage (domain.com) is 301'd to domain.com/home.html. Here's my problem/questions: 1. domain.com is indexed by Google 2. domain.com/home.html is not indexed by Google 3. both domains have some healthy linking 4. Is the fact that domain.com/home.html impacting rankings? 5. How do carefully handle this situation (ex. redirect domain.com/home.html back to domain.com?) 6. See the attached jpeg for a visual representation of my debacle. hcIiPAs
Intermediate & Advanced SEO | | rhoadesjohn0 -
.htaccess question/opinion/advice needed
Hello, I am trying to achieve 3 different things on my .htaccess I just want to make sure I am doing it the right or best way because I don't have much experience working on this kind of files. I am trying to: a) Redirect www.mysite.com/index.html to www.mysite.com so I don't get a duplicate content/tag error. b) Redirect mysite.com to www.mysite.com c) Get rid of the file extensions; www.mysite.com/stuff.html to www.mysite.com/stuff This is the code that I'm currently using and it seems to work fine, however I would like someone with experience to take a look so I can avoid internal server errors and other kinds of issues. I grabbed each piece of code from different posts and tutorials. Options +FollowSymlinks
Intermediate & Advanced SEO | | Eblan
RewriteEngine on Index Rewrite RewriteRule ^index.(htm|html|php) http://www.mysite.com/ [R=301,L] RewriteRule ^(.*)/index.(htm|html|php) http://www.mysite.com/$1/ [R=301,L] RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME}.html -f
RewriteRule ^(.*)$ $1.html Options +FollowSymlinks
RewriteEngine on
Rewritecond %{http_host} ^mysite.com [nc]
Rewriterule ^(.*)$ http://www.mysite.com/$1 [r=301,nc] Thanks a lot!0 -
Why are new pages not being indexed, and old pages (now in robots.txt) remain in the index?
I currently have a site that was recently restructured, causing much of its content to be reposted, creating new URL's for each page. To avoid duplicates, all of the existing pages were added to the robots file. That said, it has now been over a week - I know Google has recrawled the site - and when I search for term X, it is stil the old page that is ranking, with the new one nowhere to be seen. I'm assuming it's a cached version, but why are so many of the old pages still appearing in the index? Furthermore, all "tags" pages (it's a Q&A site, like this one) were also added to the robots a few months ago, yet I think they are all still appearing in the index. Anyone got any ideas about why this is happening, and how I can get my new pages indexed?
Intermediate & Advanced SEO | | corp08030 -
Multiple cities/regions websites - duplicate content?
We're about to launch a second site for a different, neighbouring city in which we are going to setup a marketing campaign to target sales in that city (which will also have a separate office there as well). We are going to have it under the same company name, but different domain name and we're going to do our best to re-write the text content as much as possible. We want to avoid Google seeing this as a duplicate site in any way, but what about: the business name the toll free number (which we would like to have same on both sites) the graphics/image files (which we would like to have the same on both sites) site structure, coding styles, other "forensic" items anything I might not be thinking of... How are we best to proceed with this? What about cross-linking the sites?
Intermediate & Advanced SEO | | webdesignbarrie0 -
While SEOing .com.au websites should I submit blogs and PRs only in .com.au Blogging / PR sites?
While SEOing .com.au, websites I am submitting PRs in sites like prweb.com, pr.com, prlog.com etc. Is that the right way or should I submit these PRs in Australian PR sites only (.com.au)?
Intermediate & Advanced SEO | | KS__0 -
Best Structure for Multi-Language/International Website
We are getting ready to do a total redsign of our website, which is a multi-language global website (www.hurco.com). Today we use an ip address lookup to determine country of origin and redirect to say hurco.de for Germany. The main reason for this was that our German division was afraid that their potential customers were going to the hurco.com site and seeing product that was not available to them. Is there a better way from an SEO standpoint to structure our website? Should we have all hurco.com traffic goto a country selection page and let them go there manually? Other good practices we should follow? Would you structure the entire site as //www.hurco.com/en-us or /en-canada (language and country) and then have all international domains 301 redirect to the proper one?
Intermediate & Advanced SEO | | fassnachtp0