Development site accidentally crawled - Will this cause problems?
-
We are currently developing a new version of our website and to make it easy to access for all team members, we just set it up on a server accessible via a publicly accessible domain name (ie devsite.com). There has been no SEO and no links created to this site, or so I thought.
Recently, I found out that Google somehow found its way to this development site and has been indexing the pages! I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain.
I immediately created a robots.txt file to disallow access to the entire development domain. My fear is that there may be some duplicate content penalty if Google sees that the content that is on our new site (once it goes live and is pushed to our REAL domain name) was previously indexed on our test domain.
We're slated to launch in 2-3 weeks. Is there anything else I should do? Should I even be worried? I'm probably a bit paranoid, but given the amount of time and effort that has gone into this new site, I love any advice or thoughts.
Thank You!
-
Great Answer, thanks Phil! One follow-up question:
In my robots.txt for the development site, I have the following:
User-agent: *
Disallow: /
Is this the correct configuration for the robots.txt file to accomplish what I want, that being removing the entire site from being crawled and from the exiting index? Or should I be configuring it differently?
Also, good tip on Webmaster Tools. I'll be request removal there as well.
-
I don't even worry about that anymore. I let Google see me build out a site anyway. I used to worry about that, but not anymore.
"I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain."
They probably came to the server and hit every site on it.
-
Setting a Robots.txt file for the Dev Site to be No index was a correct response. You can also add a No index no follow meta tag to the Dev site as well.
Another step you can take is to set up a Google Webmaster Tools account for the Dev site and block there as well.
Some dev sites are placed behind a firewall or require a sign on to access, this process can block google as well.
The risks you have is essentially creating an entire duplicate of your current website. Google will always try and crawl everything it can on the net regardless of Noindex tags. No index simply means please dont place in your index. It is important to remember that there are other Search Engines out there besides Google, Bing/yahoo, Ask, Blekko, etc... and all do not automatically honor the Noindex no follow tag. So any secure pages or documents should be just that - secured.
If those pages are no longer in the index, and are not security or confidential in nature I wouldn't worry too much.
- Phil G
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website redesign- change of server . What to do with old site? Keep for a while or delete right away?
Hey Mozzers, Two days ago, we redesigned our website and changed the server at the same time to get faster loading times. Here is what we have done. The old site was hosted on ipage, new site with a new design hosted on UPCLOUD. We changed the A record to the new server, uploaded a new site, submitted a new sitemap to Google Search console, 301 redirected all old URLs to new ones, most have changed a bit. Old URLs were ending with " .html "the new ones do not have that at the end. Submitted AMP pages to Google as well. Now here is my question. Should we delete the old site completely from ipage or should we keep it for a while? Google has indexed the new URLs that were created with the redesign, these URLs did not exist on the old site. But it still shows most of the old URLs on SERPs (these are URLs that have been 301 redirected to a new equivalent page) I understand 2 days is not very long for Google to get everything right, but I am not sure what we should do with the old site? Keep it or get rid of it to help Google index the new one only. FYI every single old URL that appears on Google search when clicked on will take you to the right place, we made sure there are no 404s at all. As this is very important to our business and we get most of it from Google I want to make sure we do it right for SEO purposes. The agency that designed the site did not really know the answer to that question, as they do not have SEO specialists. Please help, any input you might have will be greatly appreciated.
Web Design | | Davit19850 -
Thinking about redesigning site to reduce bounce rate - have a couple of questions
BACKGROUND Im looking at redesigning the website for a creative consultancy to improve the user experience. The website is mainly an image portfolio along with “press”, “our services”, “about us”, “contact” pages. I originally designed the website a few years ago and when we did, we wanted to make the image portfolio the most important feature. So we made it a full screen JS image slider with lazy loading of images, so that there are about 40 full screen images on the homepage that rotate. From a users point of view i still feel this is the best system as it very quickly allows them to browse the portfolio, which we looking for a creative consultancy is their UPS (unique selling point). The site has a very strong backlink profile compared to its competitors in the SERPS it has about 20-25% increase in PA and DA. But our site has been slipping down the rankings in recent years / months. From spot 1-2 to about spot 5-7. MY HYPOTHESIS I think that the reason the site may be dropping back in the SERP is that although its a very usable site, all its portfolio information is “too easy to find / view” and results in a user coming to our site, seeing everything they need to see, then bouncing back to the SERP. Our site has a bounce rate of 40-60%. Where as on competitors sites, their “portfolio” is a separate page off the homepage, so a users has to click through to a separate page, and even if they don't like the design content of the portfolio it doesn't get logged as a bounce. MY QUESTION Does bounce rate affect SERP ranking ? Could the sites SERP performance be improved by redesigning the site to put the portfolio on a separate page so a user would have to click through to it, if that would get the bounce rate down, would the site see a benefit even if people still clicked back to the SERP results eventually after seeing our portfolio, even though it wasn't a true 1 page bounce ? Dose time on site affect SERP ranking ? Is there a way i can see a competitor's bounce rate ? Would welcome any other thoughts inputs on this matter.
Web Design | | sl_pa0 -
Why is google still crawling my old website pages?
Why is google still looking at my old indexed pages and not my new index. ? Why are they crawling my old website links when none of them are available? How do I overcome these problems?
Web Design | | optimalspaces0 -
How does adding ecommerce to a site affect SEO? What are the negative and what are the positives?
We are thinking of adding ecommerce to our website as a service to our customers. We generate most of our leads through online quote requests but heard that it may be beneficial to our SEO if we add ecommerce for a few products. Is this true? Does anyone have tips on best and worst SEO ecommerce practices?
Web Design | | TeguarMarketing0 -
Moz crawl showing up ?s=keyword pages as errors
Hi all, Hoping someone can she some light on a fix with ref to wordpress and the search function it uses as Moz is craling some pages which reference the search domain.com/?s=keyword Errors showing up are duplicate pages, descriptions and titles. The search function is not important on this site and I have tried to use a plugin which disables the search page which it does but these errors still show up. Can anyone assist as this is the final piece of the puzzle and then we're down to 0 issues on the site.
Web Design | | wtfi0 -
How important is w3c validation for mobile sites???
So mobile sites are all the rave, but how many are doing it correctly and with all the different options which is correct or the best? For example I have a guy telling me that the mobile site must validate here http://validator.w3.org/mobile/ or here http://ready.mobi/launch.jsp?locale=en_EN However I have run many so called mobile sites like nike (m.nike.com) and those built by dudamobiles and all dramatically fail the above tests! Responsive is another key element of web design and the guys at twitter came up with bootstrap, so I ran these sites through the above validators and all have failed. I take this site as an example from ilovebootstrap.com, please note this is not my site but was top of thelist on here. Mobi Ready 2 / 5 - result poor mobile experience Results from google pagespeed Mobile 62 / 100 Desktop 83 / 100 So while it looks good on mobile devices it does not score well If you look at the google site: http://www.howtogomo.com/en-gb/d/why-get-mo/ The case studies listed all fail the validation tests, so my question is is it worth getting our mobile sites validated and will this affect rankings?
Web Design | | iprosoftware0 -
Looking for a developer with Volusion experience
We are looking for experienced developer that specializes in Volusion sites. Thanks in advance! Tony 714-744-1926
Web Design | | OCFurniture0 -
Is anyone here managing or doing SEO for a site using GoECart?
We are preparing to update/migrate to a new ecommerce platform. We are in the process of choosing right now. One of the things we know we want is faceted navigation, but I am well aware of the problems this presents for SEO. Are any of you amazing people here using, managing or have experience with GoECart? I am interested to know your feedback, particularly from an SEO viewpoint. Thanks in advance! Dana
Web Design | | danatanseo0