Google indexing staging / development site that is redirected...
-
Hi Moz Fans! - Please help.
We had a acme.stagingdomain.com while a site was in development, when it went live it redirected (302) to acmeprofessionalservices.com (real names redacted!!)
no known external links to staging site
although staging site url has been emailed from Google Apps(!!!)
now found that staging site is in the index even though it redirects to the proper public site.
and some (but not all) of the pages are in the index too. They all redirect to the proper public site when visited.
It is convenient to have a redirect from the staging site to the new one for the team, Chrome etc. remember frequently visited sites. Be a shame to lose that.
Yes, these pages can be removed using webmaster tools.
But how did they get in the index to start with?And if we're building a new site, and a customer has an existing site is there a danger of duplicate content etc. penalties caused by the staging site?
We had a similar incident recently when a PDF that was not linked anywhere on the site appeared in the index. The link had been emailed through Google Apps, and visited in Chrome, but that was it.
So 3 questions.
Why is the staging site still in the index despite the redirects?
How did they get in the index in the first place?
Will the new staging site affect the rank of the existing site, eg. duplicate content penalties?
-
Hi There
1. It could still be in the index because they are 302 redirect and not 301. 302 is temporary, and therefore Google may not de-index those URLs. It also takes time. I've seen Google take months to noindex redirecting URLs. Also, make sure you are not blocking crawling of the dev site, or Google will not see the redirects.
2. I am not sure how they got there to begin with. I pretty much always can find some sort of error - maybe someone tweeted a staging URL, maybe crawling wasn't blocked, maybe there was one link to staging from the live site etc etc. Regardless - somehow Google crawled it
To prevent this in the future always block crawling of staging servers well before you ever put anything on them.
3. Usually Google tries to sort this out. They won't give you a penalty for "technical" duplicate content (penalties are more for "malicious" duplicate content ie: stealing people's content). So you won't get penalized, but the more you can help Google out by sorting it out, the more time Google can spend crawling the correct site etc.
What I would do now is, if you do want the staging URLs to redirect (which might not be the best solution if you want to ever go back and work on the staging server again) - but if you do, use 301 redirects and make sure you are allowing crawling of the staging site. Keep it registered in webmaster tools and this way you can monitor the indexation levels.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved URL dynamic structure issue for new global site where I will redirect multiple well-working sites.
Dear all, We are working on a new platform called [https://www.piktalent.com](link url), were basically we aim to redirect many smaller sites we have with quite a lot of SEO traffic related to internships. Our previous sites are some like www.spain-internship.com, www.europe-internship.com and other similars we have (around 9). Our idea is to smoothly redirect a bit by a bit many of the sites to this new platform which is a custom made site in python and node, much more scalable and willing to develop app, etc etc etc...to become a bigger platform. For the new site, we decided to create 3 areas for the main content: piktalent.com/opportunities (all the vacancies) , piktalent.com/internships and piktalent.com/jobs so we can categorize the different types of pages and things we have and under opportunities we have all the vacancies. The problem comes with the site when we generate the diferent static landings and dynamic searches. We have static landing pages generated like www.piktalent.com/internships/madrid but dynamically it also generates www.piktalent.com/opportunities?search=madrid. Also, most of the searches will generate that type of urls, not following the structure of Domain name / type of vacancy/ city / name of the vacancy following the dynamic search structure. I have been thinking 2 potential solutions for this, either applying canonicals, or adding the suffix in webmasters as non index.... but... What do you think is the right approach for this? I am worried about potential duplicate content and conflicts between static content dynamic one. My CTO insists that the dynamic has to be like that but.... I am not 100% sure. Someone can provide input on this? Is there a way to block the dynamic urls generated? Someone with a similar experience? Regards,
Technical SEO | | Jose_jimenez0 -
URL structuring / redirect question
Hi there, I have a URL structuring / redirect question. I have many pages on my site but I set each page up to fall under one of two folders as I serve two unique markets and want each side to be indexed properly. I have SIDE A: www.domain/FOLDER-A.com and SIDE B: www.domain/FOLDER-B. The problem is that I have a page for www.domain.com and www.domain/FOLDER-A/page1.com but I do NOT have a page for www.domain/FOLDER-A. The reason for this is that I've opted to make what would be www.domain/FOLDER-A be www.domain.com and act the primary landing page the site. As a result, there is no page located at www.domain/FOLDER-A. My WordPress template (Divi by Elegant Themes) forced me to create a blank page to be able to build off the FOLDER-A framework. My question is that given I am forced to have this blank page, do I leave it be or create a 302 or 307 redirect to www.domain.com? I fear using a 301 redirect given I may want to utilize this page for content at some point in the future. This isn't the easiest post to follow so please let me know if I need to restate the question. Many thanks in advance!
Technical SEO | | KurtWSEO0 -
Example of Google Indexing my Feedburner Links
As you can see, there are 2 results for the same page. One is the correct page URL, the other has the Feedburner parameters at the end: http://www.thewebhostinghero.com/articles/improving-user-engagement-with-the-right-blog-commenting-system.html http://www.thewebhostinghero.com/articles/improving-user-engagement-with-the-right-blog-commenting-system.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+thewebhostinghero+(TheWebHostingHero.com) Can this cause duplicate content issues? Can I prevent Google from indexing my Feedburner links? My Feedburner settings are already set to noindex, what else can I do?!? 22cfThX.png
Technical SEO | | sbrault740 -
Redirecting root to /default.aspx
Hello, I have a client who's home page redirects to /default.aspx - what are the possible SEO impacts of this? As the home page redirects to /default.aspx and does not load under both there does not seem to be a duplicate content issue. Also the redirect should carry over most of the link juice from the home page to /default.aspx therefore are there any negative SEO "side effects" of this set-up? Thanks in advance!
Technical SEO | | RikkiD220 -
How should i knows google to indexed my new pages ?
I have added many products in my ecommerce site but most of the google still not indexed yet. I already submitted sitemap a month ago but indexed process was very slow. Is there anyway to know the google to indexed my products or pages immediately. I can do ping but always doing ping is not the good idea. Any more suggestions ?
Technical SEO | | chandubaba1 -
Does http://my.dudamobile.com/ Effect SEO
Hi, Hope everyone is enjoying the new year! I was wondering if converting your desk top website to a mobile one, example via http://my.dudamobile.com/, has any negative effects on SEO. Did it effect your site? Do you recommend doing it? Does it effect links? When people link to your desk top URL does that authority carry to the mobile, or would it be better if they link to the mobile (m.website.com) URL? Is http://my.dudamobile.com/ a good choice? Any feedback, as always, is greatly appreciated! Thanks Jimmy
Technical SEO | | jimmy02250 -
Photography Sites with Same Developer - Why Is One Ranking & Other Not?
I'm currently confused about the difference in ranking between two competing sites, created by the same agency. http://jmayphoto.com/index2.php#!/home (302 redirected from http://jmayphoto.com...yeah) is not ranking well, and I'm not surprised. However, competitor http://www.shanrenee.com/ is ranking within the top 5 spots for a primary target keyword (dallas wedding photographer) and I don't understand how it's doing so well. I definitely see differences, but not enough to explain how Shan Renee is one page. What am I missing?
Technical SEO | | BrittanyHighland0 -
Can I turn off Google site links?
I thought at one time I had turned off the option to have Google sitelinks. I did this so that each of our pages that had a strong presence would occupy a unique slot on the first and second page of Google. This was important to us as we were battling some reputation management issues and trying to push out negative listings from the front page. Recently I noticed sitelinks were back up and when going into Google Webmaster Tools, I could figure out how to opt out of them. Any suggestions?
Technical SEO | | BRConsulting0