Development site accidentally crawled - Will this cause problems?
-
We are currently developing a new version of our website and to make it easy to access for all team members, we just set it up on a server accessible via a publicly accessible domain name (ie devsite.com). There has been no SEO and no links created to this site, or so I thought.
Recently, I found out that Google somehow found its way to this development site and has been indexing the pages! I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain.
I immediately created a robots.txt file to disallow access to the entire development domain. My fear is that there may be some duplicate content penalty if Google sees that the content that is on our new site (once it goes live and is pushed to our REAL domain name) was previously indexed on our test domain.
We're slated to launch in 2-3 weeks. Is there anything else I should do? Should I even be worried? I'm probably a bit paranoid, but given the amount of time and effort that has gone into this new site, I love any advice or thoughts.
Thank You!
-
Great Answer, thanks Phil! One follow-up question:
In my robots.txt for the development site, I have the following:
User-agent: *
Disallow: /
Is this the correct configuration for the robots.txt file to accomplish what I want, that being removing the entire site from being crawled and from the exiting index? Or should I be configuring it differently?
Also, good tip on Webmaster Tools. I'll be request removal there as well.
-
I don't even worry about that anymore. I let Google see me build out a site anyway. I used to worry about that, but not anymore.
"I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain."
They probably came to the server and hit every site on it.
-
Setting a Robots.txt file for the Dev Site to be No index was a correct response. You can also add a No index no follow meta tag to the Dev site as well.
Another step you can take is to set up a Google Webmaster Tools account for the Dev site and block there as well.
Some dev sites are placed behind a firewall or require a sign on to access, this process can block google as well.
The risks you have is essentially creating an entire duplicate of your current website. Google will always try and crawl everything it can on the net regardless of Noindex tags. No index simply means please dont place in your index. It is important to remember that there are other Search Engines out there besides Google, Bing/yahoo, Ask, Blekko, etc... and all do not automatically honor the Noindex no follow tag. So any secure pages or documents should be just that - secured.
If those pages are no longer in the index, and are not security or confidential in nature I wouldn't worry too much.
- Phil G
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will There Be Much Impact When Moving Site To New Root Folder?
Hi, ok so I have a pretty big site that is located on my sever /root/current-folder/. I want to rebuild the site completely as it's using software that is out of date and not our main focus anymore (OpenCart). We want to move to a Wordpress platform, but want to have as little impact on the SEO as possible. Our current strategy is: List all URLs/Titles/Meta indexed with Google on current site Create new folder on the server /root/new-folder/ My question is... if I move to a new folder on the server (same TLD) and then re-route the TLD to go to this new folder, will there be more of an impact on SEO that if I start a fresh in the current folder? Thanks
Web Design | | Easigrass0 -
Site Migration due to Corporate Acquisition
Hey everyone, Wanted to check-in on something that I've been thinking way too much about lately. I'll do my best to provide background, but due to some poor planning, it is rather confusing to wrap your head around. There are currently three companies involved, Holding Corp (H Corp) and two operating companies, both in the same vertical but one B2B and the other is B2C. B2C corp has been pushed down the line and we're focusing primarily on H Corp and B2B brand. Due to an acquisition of H Corp and all of it's holdings, things are getting shuffled and Ive been brought in to ensure things are done correctly. What's bizarre is H Corp and it's web property are the dominant authority in SERPs for the B2B brand. As in B2B brand loses on brand searches to H Corp, let alone any product/service related terms. As such, they want to effectively migrate all related content from H Corp site to B2B brand site and handover authority as effectively as possible. Summary: Domain Migration from H Corp site to B2B Brand site. Ive done a few migrations in my past and been brought in to recover a few post-launch so I have decent experience and a trusted process. One of my primary objectives initially is change as little as possible with content, url structure (outside the root) etc so 301s are easy but also so it doesn't look like we're trying to play any games. Here's the thing, the URL structure for H Corp is downright bad from both a UX perspective and a general organizational perspective. So Im feeling conflicted and wanted to get a few other opinions. Here are my two paths as I see and Id love opinions on both: stick with a similar URL structure to H Corp through the migration (my normal process) but deviate from pretty much every best practice for structuring URLs with keywords, common sense and logic. Pro: follow my process (which has always worked in the past) Con: don't implement SEO/On-page best practices at this stage and wait for the site redesign to implement best practices (more work) Implement new URL structure now and deviate from my trusted process. Do you see a third option? Am I overthinking it? Other important details: B2B brand is under-going a site redesign, mostly aesthetic but their a big corporation and will likely take 6-9 months to get up. Any input greatly appreciated. Cheers, Brent
Web Design | | pastcatch1 -
Incorporating Spanish Page/Site
We bought an exact match domain (in Spanish) to incorporate with regular website for a particular keyword. This is our first attempt at this, and while we do have Spanish speaking staff that will translate/create a nice, quality page, we're not going to redo everything in Spanish page. Any advice on how to implement this? Do I need to create a whole other website in Spanish? Will that be duplicate content if I do? Can I just set it up to show the first page in Spanish, but if they click on anything else it redirects to our site? I'm pretty clueless on this, so if anything I've suggested is off-the-wall or a violation, I'm really just spit-balling, trying to figure out how to implement this. Thanks, Ruben
Web Design | | KempRugeLawGroup0 -
Large Global Site Structure
Hi, I have a question about the advised structure for a website that I'm currently building. It's a large international brand with it's main office in the UK. The main website is the .com but there is a growing international franchisee network. I've built the .com site on Wordpress but I'm not sure if I the best way forward would be to create each international website on a separate hosted site or just include it in the .com Wordpress structure using the The WordPress Multilingual Plugin. So to sum up... should I build the entire global network on one domain and then use WPML plugin or should I build separate websites for each International franchisee? Hope some one can educate me on the best route to take. Thanks Moz Community
Web Design | | SeoSheikh0 -
Show new mobile site to 60% users & old mobile site to 40% users
Hi, We are planning to show new mobile site to 60% users & old mobile site to 40% users. We will show the old site to google crawler. Our old site has some interlinking through footer & content whereas the new site does not has it. We wanted to do this since our new site does not supports some browsers. Will there be an issue with Google on showing the site like this. The mobile site & desktop site will have same url across devices & browsers. Regards
Web Design | | vivekrathore0 -
Affected by the Penguin 2.0...Lost Rankings!! Which one from the following causes did that?
Hi Moz Community, we lost our rankings for almost all of our keywords due to penguin 2.0 I'm trying to rectify the situation by getting our rankings back and better. Here are the potential causes for our rankings drop. 1. My Company has the exact same site (in English) on different country-specific domains. companyname.com, companyname.ch, companyname.ca, companyname.nl etc.. We are using link rel="alternate" hreflang="en" href="http://companyname.com" **Would this solve the duplicate content issue? ** 2. All the international domains have footer links in the homepages, pointing to our main .com site. Google webmaster tools shows that every page on these international domains as a backlink source. Would that be a problem? Should I ask our web designers to remove those footer links from the international domains? 3. In the past, our company had a partnership with forum...Like the forum promotes our company's services. I don't know how did that forum promoted our site, But Google webmaster tools shows that we have some 25K backlinks from that forum!!! But when I try to visit those backlinks, none of those pages have any links to our site....Could they be hidden? Or Some thing wrong with GWT? I'd appreciate your valuable suggestions to help us understand the situation better... Thank you.
Web Design | | hninformer0 -
SEO tricks for a one page site with commented html content
Hi, I am building a website that is very similar to madebysofa.com : means it is one page site with entire content loaded (however are commented in html) and by clicking on sections it modify the DOM to make specific section visible. It is very interesting from UX point of view but as far as I know, since this way most of my content is always commented and hidden from crawlers, I will loose points regarding SEO. Is there any workaround you can recommend or you think sites like madebysofa.com are doomed to loose SEO points by nature? Best regards,
Web Design | | Ashkan10 -
How many sites on one hosting account?
How many sites is safe to house on one hosting provider? I use BlueHost and they advertise unlimited domains, but I'm not sure what the negative side effects might be from hosting too many on one hosting service. If it matter at all, I'm using WordPress to build my sites. Pros and Cons?
Web Design | | leafndrop0