Development site accidentally crawled - Will this cause problems?
-
We are currently developing a new version of our website and to make it easy to access for all team members, we just set it up on a server accessible via a publicly accessible domain name (ie devsite.com). There has been no SEO and no links created to this site, or so I thought.
Recently, I found out that Google somehow found its way to this development site and has been indexing the pages! I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain.
I immediately created a robots.txt file to disallow access to the entire development domain. My fear is that there may be some duplicate content penalty if Google sees that the content that is on our new site (once it goes live and is pushed to our REAL domain name) was previously indexed on our test domain.
We're slated to launch in 2-3 weeks. Is there anything else I should do? Should I even be worried? I'm probably a bit paranoid, but given the amount of time and effort that has gone into this new site, I love any advice or thoughts.
Thank You!
-
Great Answer, thanks Phil! One follow-up question:
In my robots.txt for the development site, I have the following:
User-agent: *
Disallow: /
Is this the correct configuration for the robots.txt file to accomplish what I want, that being removing the entire site from being crawled and from the exiting index? Or should I be configuring it differently?
Also, good tip on Webmaster Tools. I'll be request removal there as well.
-
I don't even worry about that anymore. I let Google see me build out a site anyway. I used to worry about that, but not anymore.
"I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain."
They probably came to the server and hit every site on it.
-
Setting a Robots.txt file for the Dev Site to be No index was a correct response. You can also add a No index no follow meta tag to the Dev site as well.
Another step you can take is to set up a Google Webmaster Tools account for the Dev site and block there as well.
Some dev sites are placed behind a firewall or require a sign on to access, this process can block google as well.
The risks you have is essentially creating an entire duplicate of your current website. Google will always try and crawl everything it can on the net regardless of Noindex tags. No index simply means please dont place in your index. It is important to remember that there are other Search Engines out there besides Google, Bing/yahoo, Ask, Blekko, etc... and all do not automatically honor the Noindex no follow tag. So any secure pages or documents should be just that - secured.
If those pages are no longer in the index, and are not security or confidential in nature I wouldn't worry too much.
- Phil G
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Recovering organic traffic and Google rankings post-site-crash
Hi everyone, we had a client's Wordpress website go down about 2 weeks ago and since then organic traffic has basically plummeted. We haven't identified exactly what caused the crash, but it happened twice in one week. We spent a lot of time optimizing the site for organic SEO, improving load times, improving user experience, improving the website content, improving CTR, etc. Then one morning we get a notification from our uptime monitoring service that the site was down, and upon further inspection we believe it may have been compromised. The child theme that the website was using, all of the files were deleted and/or blank. We reverted the website to a previous backup, which fixed the problem. Then, a few days later, the same exact thing happened, only this time the child theme files were missing after the backup was restored. We've since re-installed and reconfigured the child theme, changed all passwords (Wordpress, FTP, hosting, etc.), and we're looking into changing hosting providers in the very near future. The site uses the Yoast Wordpress SEO plugin, which has recently been reported as having some security flaws. Maybe that was the cause of the problem. Regardless, the primary focus right now is to recover the organic traffic and Google rankings that we've worked so hard to improve over the past few months up until this disaster occurred. The client is in a very competitive niche and market, so I'm pretty frustrated that this has happened after we were making such great progress, Since the website went down, organic search traffic has decreased by 50%. The site and all internal pages are loading properly again (and have been since the second time the website went down), but Google Webmaster Tools is still reporting a number of pages as "not found" witht he crawl dates as early as this past weekend. We've marked all errors as "fixed", and also re-submitted the Sitemaps in Google Webmaster Tools. The website passes the "mobile-friendly" tests, received A and B grades in GTMMetrix (for whatever that's worth), and still has the same original Google Maps rankings as before. The organic traffic, however, and organic rankings on Google have seen a pretty dramatic decrease. Does anyone have any recommendations when it comes to recovering a website's authority and organic traffic after it's experienced some downtime?
Web Design | | georgetsn0 -
Major URL changes in new site launch
Hey Guys - we recently launched a new website for a client. Prior, all of their URLs were dynamic via an old-school Cold Fusion CMS. We basically had to rewrite 90% of the sites URLs (site is like 300 pages). The new URLs are SEO friendly and the on-page SEO is strong; but the page rank/authority is starting from scratch from these pages and placement has decreased more most of the new pages with competitive keywords. We set up all of the 301 redirects properly and are actively monitoring in Google Webmaster Tools. **Anything else I can do to lessen the pain and get these pages higher page rank/authority sooner rather than later?**Thanks for all of your help.
Web Design | | NobleStudios0 -
Migrating a site to Wordpress
I've recently been converting our old website to a wordpress based website and been working on the new version of the site on a subdomain. Now at the stage when I am getting ready to let the site go live and just wondering exactly how to do this so I have minimal downtime? Looking in the wordpress control panel there is the setting to enter the address of the site if you want it to be different from the directory it has been installed within - is this a good idea (i.e. is it stable if I do this? good for seo, bad for seo or makes no difference?)? or should I manually install everything in the root myself (if I do this is there a way to direct people to the temp version of the site on the subdomain? Any tips, do and don't s would be appreciated as I want to do this right!
Web Design | | Jon-C0 -
Looking for feedback on our nonprofit site
I work for a nonprofit org which of course means a low budget and paying out of pocket for things (such as training). Our current website is done by a 3rd party vendor and although it looks nice, we can't make any changes to it without paying for it. (We can only upload documents). I'm wondering if anyone in this group will give their feedback on the site in terms of SEO and recommend a platform that would be relatively easy for a small shop to manage. Our site is www.coastalcommunityfoundation.org Thanks in advance
Web Design | | TinaA0 -
I've set up my own site which is still fairly new but I'm a bit concerned that there is a bloackage SEO wise somewhere because when I try to crawl the site on SEOmoz it only crawls one page.
I'm really baffled and none of my research has shed much light on it. My url is www.emporiumofmanliness.co.uk I'd really appreciate any help! Thanks
Web Design | | JoshED0 -
Website Blog causes duplicate pages
Hello, I added a blog to my website, which is hosted at weebly. I was told this would drive traffic but I have actually fallen way, way down in Alexa rankings. When I ran a campaign here, the results show over a 100 errors, all to do with the website blog. It states they are duplicate pages and titles. I dont see a way to rename the pages. Am I better off getting rid of the blog? Thanks
Web Design | | Gardengirl0 -
Critique a site I'm working on please
Hi guys, I have a list of things to do on this website that is longer than my arm but thought I would gather a few more thoughts on the site before moving on further with it. The site is here. It went live a couple of days ago for live testing. I have a news section coming very soon at theworkplacedepot.co.uk/news - it is in live testing now. One glaring issue for me is that some of the products simply aren't "buyable". What I mean by that is that the options are too vague and too vast - as an outsider I wouldnt have a clue what I was buying! Other issues: The banner at the bottom of the HP is a test - its design will be changing. The sitemap has some issues and will be addressed. There is no area yet to "log-in". However, critique for me pretty please! 🙂
Web Design | | MattJanaway0 -
Can i do this? Will Google penalize me?
I have a page for a Criminal Defense Attorney and i set up a list of the type of criminal charges he is certified to deal with. I wanted to use title tags and put the Keyword "Miami Criminal Defense Attorney" & "Miami Traffic Defense Lawyer"... My question is will Google penalize me for plugging the same Key words over and over on the title tag for each ?? CHECK THE IMAGE to see what I'm talking about... thanks guys. x97dl
Web Design | | marig0