Development site accidentally crawled - Will this cause problems?
-
We are currently developing a new version of our website and to make it easy to access for all team members, we just set it up on a server accessible via a publicly accessible domain name (ie devsite.com). There has been no SEO and no links created to this site, or so I thought.
Recently, I found out that Google somehow found its way to this development site and has been indexing the pages! I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain.
I immediately created a robots.txt file to disallow access to the entire development domain. My fear is that there may be some duplicate content penalty if Google sees that the content that is on our new site (once it goes live and is pushed to our REAL domain name) was previously indexed on our test domain.
We're slated to launch in 2-3 weeks. Is there anything else I should do? Should I even be worried? I'm probably a bit paranoid, but given the amount of time and effort that has gone into this new site, I love any advice or thoughts.
Thank You!
-
Great Answer, thanks Phil! One follow-up question:
In my robots.txt for the development site, I have the following:
User-agent: *
Disallow: /
Is this the correct configuration for the robots.txt file to accomplish what I want, that being removing the entire site from being crawled and from the exiting index? Or should I be configuring it differently?
Also, good tip on Webmaster Tools. I'll be request removal there as well.
-
I don't even worry about that anymore. I let Google see me build out a site anyway. I used to worry about that, but not anymore.
"I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain."
They probably came to the server and hit every site on it.
-
Setting a Robots.txt file for the Dev Site to be No index was a correct response. You can also add a No index no follow meta tag to the Dev site as well.
Another step you can take is to set up a Google Webmaster Tools account for the Dev site and block there as well.
Some dev sites are placed behind a firewall or require a sign on to access, this process can block google as well.
The risks you have is essentially creating an entire duplicate of your current website. Google will always try and crawl everything it can on the net regardless of Noindex tags. No index simply means please dont place in your index. It is important to remember that there are other Search Engines out there besides Google, Bing/yahoo, Ask, Blekko, etc... and all do not automatically honor the Noindex no follow tag. So any secure pages or documents should be just that - secured.
If those pages are no longer in the index, and are not security or confidential in nature I wouldn't worry too much.
- Phil G
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it Bad to Break Up A Site into Multiple Sites?
I have a big cluttered website with endless pages. It's a non-profit that has content for patients, researchers, therapists, etc.. Would it be a bad idea to turn this cluttered site into 3 or more completely different sites, each focused on their specific demographic? Or should I just figure out how to organize the one site better? Thanks for your help!!!
Web Design | | bosleypalmer0 -
Lost Rankings Late April Even Though We Have A Mobile Site
I have noticed a significant drop in rankings since late April. It is about a 30% drop in organic from Google. This is despite the fact that we launched a mobile site before the update. What gives? Any thoughts or suggestions would be appreciated.
Web Design | | inhouseseo0 -
404's and a drop in Rank - Site maps? Data Highlighter?
I managed an old (2006 design) ticket site that was hosted and run by the same company that handled our point of sale. (Think, really crappy, customer had to click through three pages to get to the tickets, etc.) In Mid February, we migrated that old site to a new, more powerful site, built by a company that handles sites exclusively for ticket brokers. (My site: TheTicketKing. - dot - com) Before migration, I set up 301's for all the pages that we had currently ranked for, and had inbound links pointing to, etc. The CMS allowed me to set every one of those landing pages up with fresh content, so I created unique content for all of them, ran them through the Moz grader before launch, etc. We launched the site in Mid February, and it seemed like Google responded well. All the pages that we had 301's set up for stayed up fairly well in rank, and some even reached higher positions, while some took a few weeks to get back up to where they were before. Google was also giving us an average of 8-10K impressions per day, compared to 3000 per day with the old site. I started to notice a slow drop in impressions in mid April (after two months of love from Google,) and we lost rank on all our non branded pages around 4/23. Our branded terms are still fine, we didn't get a message from Google, and I reached out to the company that manages our site, asking if they had any issues with their other clients. They suggested that I resubmit our sitemaps. I did, and saw everything bump back up (impressions and rank) for just one week. Now we're back in the basement with all the non branded terms once again. I realize that Google could have penalized us without giving us a message, but what got me somewhat optimistic was the fact that resubmitting our sitemaps did bring us back up for around a week. One other thing that I was working on with the site just before the drop was Google's data highlighter. I submitted a set of pages that now come back with errors, after Google seemed to be fine with the data set before I submitted it. So now I'm looking at over 300 data highlighter errors when I'm in WMT. I deleted that set, but I still get the error listings in WMT, as if Google is still trying to understand those pages. Would that have an effect on our rank? Finally I do see that our 404's have risen steadily since the migration, to over 1000 now, and the people who manage the CMS tell me that it would have no effect on rank overall. And we're going to continue to get 404's as the nature of a ticket site would dictate? (Not sure on that, but that's what I was told.) Would anyone care to chime in on these thoughts, or any other clues as to my drop?
Web Design | | Ticket_King0 -
Site with no ads hit by Page Layout update?
Hi there! Can a site that has no ads on it be hit by Google's latest Page Layout update? Can it be hit for just one or two keywords? My site (www.ink2paper.com) has a decline in Google organic traffic in early Feb so my suspicion is the Page Layout update. However I have no ads on the site. Digging into GWMT I find that it is only one or 2 keywords that seems to have taken a dive, mainly [photo paper]. I used to get around 80 imps a day for this term. Then on 6 Feb it was down to 50; 7 Feb = 34; 8 Feb just 4 impressions! I got a spike back at usual levels on 10 & 11 Feb, but since then it has been back down to only 5 or so impressions a day. [photographic paper] took a small hit at the start of February, but has nose dived since the start of April. The homepage performs well for Google organic traffic - low bounce (22%) and good ecom conversion rate (14%) - although this is likely to be largely branded traffic. I feel my site is a 'good' result for the search term [photo paper], although there is always room for improvement of course! Any suggestions as to why Google has stopped showing my site for these keywords? All help is greatly appreciated. Cheers,
Web Design | | SimonHogg
Simon0 -
Old site to new WordPress site - Client concerned about Yahoo Ranking
Hello, Back Story I have a client (law firm) who has a large .html website. He has been doing his own SEO for years and it shows. I think the only reason he reached out to a professional is because he got a huge penalty from Google last fall and fell very far down in rankings. Although, he still retains a #1 spot in Yahoo for his site for the keyword phrase he wants. I have been creating a new WordPress theme for the client and creating all new pages and updating the formatting/SEO. From the beginning I have told the client that when we delete the old site and install a new WordPress site (same domain name, but different page hierarchy) he will take a bump in the search engines until all the 301 redirects get sorted out. I told him I can't guarantee any time frame of how long the dip in SEO will last. Some sites bounce right back while others take longer. Last week, during a discussion, he tells me that if he loses his #1 ranking on Yahoo for any length of time he thinks he will go out of business. Needless to say I was a little taken back. When it comes to SEO I use best practice techniques, do my research, stay on top of trends but I never guarantee rankings when moving to a new site. I'm thinking of ways I can help elevate any type of huge SEO drop off and help the client. Here is what I was thinking of suggesting to the client and I would love some feedback. Main Question He has another domain he isn't doing anything with. It's pretty much his domain name with pc added. I was thinking about using that domain to create a simple 1-2 page WordPress website with brand new content (no duplicate content) aimed at attracting his keyword phrase. I would do as much SEO as I could with a 1-2 page site and give it a month or so to see if this smaller site can get into the top #10 in Yahoo, or higher. Then, when we move the site he will still have a website on the first page of Yahoo for his keyword phrase. I hope I explained it clearly 🙂 I would be open to any suggestions anyone may have. Thanks
Web Design | | Bill_K0 -
Site Rebuild -Larger to smaller
Hi All, We are rebuilding an existing site which has around 230 Pages (lots of content not required) down to around 20. Whats the best way to 301 redirect the pages that are going to be removed- (we wont be able to use .htaccess because we are moving to Adobe Business Catalyst) Thoughts? We are trying to preserve as much SEO value as possible.....
Web Design | | OnlineAssetPartners0 -
My site build in HTML has been badly hit this recent update and I have been toying on the idea of changing it to Wordpress
My site build in HTML has been badly hit this recent update and I have been toying on the idea of changing it to Wordpress. Would this help in my rankings? It seems the hit came only that last 2-4 days when business become much quieter. Frankly i have no idea on why the site fell in ranking all of a sudden. Been comparing to competitors and even a friend in the same industry and nothing makes sense so far (link profile, DA, PA etc) This site has been ranking well for 3 years prior to this. 2. My site has loads of content and visitors arrive from various landpages. But by changing it wordpress, the url of most of them would probably change. What should i do? 301 redirect all of them or is there a better method?
Web Design | | rester0 -
Duplicate content on mobile sites
Hi Guys We are launching a mobile webshop later this year and have decided to use a subdomain for this. (m.domainname.xx). The content will be more or less identical with the one on the standard desktop site (domainname.xx), but im struggeling to find out if this will create dipplicate content between the mobile and desktop site. Does anyone have a solid answer for this one?
Web Design | | AndersDK0