Development site accidentally crawled - Will this cause problems?
-
We are currently developing a new version of our website and to make it easy to access for all team members, we just set it up on a server accessible via a publicly accessible domain name (ie devsite.com). There has been no SEO and no links created to this site, or so I thought.
Recently, I found out that Google somehow found its way to this development site and has been indexing the pages! I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain.
I immediately created a robots.txt file to disallow access to the entire development domain. My fear is that there may be some duplicate content penalty if Google sees that the content that is on our new site (once it goes live and is pushed to our REAL domain name) was previously indexed on our test domain.
We're slated to launch in 2-3 weeks. Is there anything else I should do? Should I even be worried? I'm probably a bit paranoid, but given the amount of time and effort that has gone into this new site, I love any advice or thoughts.
Thank You!
-
Great Answer, thanks Phil! One follow-up question:
In my robots.txt for the development site, I have the following:
User-agent: *
Disallow: /
Is this the correct configuration for the robots.txt file to accomplish what I want, that being removing the entire site from being crawled and from the exiting index? Or should I be configuring it differently?
Also, good tip on Webmaster Tools. I'll be request removal there as well.
-
I don't even worry about that anymore. I let Google see me build out a site anyway. I used to worry about that, but not anymore.
"I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain."
They probably came to the server and hit every site on it.
-
Setting a Robots.txt file for the Dev Site to be No index was a correct response. You can also add a No index no follow meta tag to the Dev site as well.
Another step you can take is to set up a Google Webmaster Tools account for the Dev site and block there as well.
Some dev sites are placed behind a firewall or require a sign on to access, this process can block google as well.
The risks you have is essentially creating an entire duplicate of your current website. Google will always try and crawl everything it can on the net regardless of Noindex tags. No index simply means please dont place in your index. It is important to remember that there are other Search Engines out there besides Google, Bing/yahoo, Ask, Blekko, etc... and all do not automatically honor the Noindex no follow tag. So any secure pages or documents should be just that - secured.
If those pages are no longer in the index, and are not security or confidential in nature I wouldn't worry too much.
- Phil G
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site redesign - 301 Redirects
I've just overhauled a website, leaving lots of former posts in the dust. I've set up a 404 redirect to the home page so that if anyone goes to one of those old pages they land on the home page instead of a dead 404. But, there's a couple urls from the old site I'd prefer to redirect to similar pages. These urls have forward slashes and I don't know how to get the slashes in when I copy it over to the new site. This is probably something easy, but I'm baffled. This www.lawbarron.com/personal-injury/whip-its-nitrous-oxide/ becomes this when I copy it www.lawbarron.com/personal-injury-whip-its-nitrous-oxide Can someone help me out?
Web Design | | julie-getonthemap0 -
Is this spammy/panda problem?
We have a property site and have many area pages with listings of properties. For example, one page may have up to 30 properties listed, with 100 words of description on that listing page for that property and then you click 'more info' to get to the page of that property with maybe 200 words in total. I want to add bullet points for each property on the area page but Im worried that it may be seen by google as spammy even though its usefull to the client. For example, if I had 30 properties on a page, and 28 of them said next to each picture.. Property Type: Shared, Open Plan, Single Bed Would that be a problem for google?
Web Design | | nick-name1230 -
Footer links on my site... bad for passing page rank?
i've been told that it is possible that google discounts the weight or page rank passed in footer links of websites and my website has the navigation to many of my pages in the footer of each page. My whole website is about 20 pages so each page has links to the 5 most popular pages at the top and the rest of the links are in the footer of each page. Am i losing page rank by having these links in the footer? Should i make my navigation different? I have lots of articles on my site so i thought it might be not only helpful to my readers but give my pages an seo boost if i placed in context links in the body of my articles to other pages of my site. Does this sound like a good idea? Thanks mozzers! Thanks mozzers!
Web Design | | Ron100 -
ECWID Ecommerce Sites. No Custom URLS?
Is there any way possible to be able to name product urls in website that use ECWID for their ecommerce? They have long and "dirty" urls. For example this running boards site: http://www.runningboards4less.com/general-motors#!/~/product/category=6593890&id=28043027 Isn't this hurting the overall SEO of the site? Especially product pages?
Web Design | | Atlanta-SMO0 -
Will changing product from Grouped to Simple on my magento category page affect my SEO?
Hi all, A category page on my site http://www.porcelainsuperstore.co.uk/wood-effect.html currently ranks number 3 on Google for the keyword "Wood Effect Tiles" We're currently reorganising some of our product and I would like to know if this is going to affect the SEO and ranking for the above page and keyword. The majority of products on that page are magento grouped products. I would like to change the page so that it displays only the different constituent simple products rather than the grouped products on the category page. My question is, will this have any impact on SEO? I intend on leaving all other data on the category page the same - so the metadata and the description/title etc. Any help/comments would be much appreciated! Ben
Web Design | | piazza0 -
Home page redirect - will this cause an SEO problem
Hello, We are using Wordpress to build a wiki site. The wiki plugin we're using (Wordpress Wiki lite) can only be set up on an internal page like nlpwiki(dot)org/wiki Can we redirect the home page to the /wiki subdirectory and use nlpwiki(dot)org/wiki as our home page? I've never done that, just wondering if it will be indexed as the home page or if there are any connonical issues. Thanks!
Web Design | | BobGW0 -
I have a question about converting a flash site to word press
We're changing one of our site to word press from flash and the old site had no content, so everything is going to be different. Should we try to match the page names and small amount of content and large amount of meta tags for now and slowly change them, or is it alright to change everything right now? The site has good ranking and I don't want to mess it up.
Web Design | | allstatetransmission0 -
Site-wide footer links or single "website credits" page?
I see that you have already answered this question before back in 2007 (http://www.seomoz.org/qa/view/2163), but wanted to ask your current opinion on the same question: Should I add a site-wide footer link to my client websites pointing to my website, or should I create a "website credits" page on my clients site, add this to the footer and then link from within this page out to my website?
Web Design | | eseyo0