Development site crawled
-
We just found out our password protected development site has been crawled. We are worried about duplicate content - what are the best steps to take to correct this beyond adding to robots.txt?
-
Unfortunately, robots.txt won't prevent your site from being crawled and indexed if there is a link from an external site pointing to yours. What you need to do is use
on all your development pages. I don't know how big your site is, so this may or may not be a lot of work. Do this, then after the next Google crawl, your pages will be dropped from the SERPs.
-
Thanks Stephen & Kyle! We had the site behind a login, so we're not sure how this happened. Any idea?
-
Put the site behind a login
-
Oops! That sounds unfortunate, Marcy. How did that happen?
Once you have added the correct rules to the robots.txt - I'm guessing you're using "Disallow: /" - you can request, if your development site is registered in Google Webmaster Tools, that Google remove the site from its index.
www.google.com/webmasters/tools/url-removal
Hope that helps,
K
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Has Not Recovered (Still!) from Penguin
Hello, I have a site that just keeps getting hit with a ton of bad, unnatural backlinks due to the sins from previous SEO companies they've hired. About every quarter, I have to add more bad links to their disavow file... still. Is it time to move them to a new domain? Perhaps a .net? If so, do we just completely trash the old domain & not redirect it? I've never had a client like this in the past but they still want to maintain their branded name. Thanks for your feedback!
Intermediate & Advanced SEO | | TinaMumm0 -
Is this ok for content on our site?
We run a printing company and as an example the grey box (at the bottom of the page) is what we have on each page http://www.discountbannerprinting.co.uk/banners/vinyl-pvc-banners.html We used to use this but tried to get most of the content on the page, but we now want to add a bit more in-depth information to each page. The question i have is - would a 1200 word document be ok in there and not look bad to Google.
Intermediate & Advanced SEO | | BobAnderson0 -
Need help or explanation on my site!
My site has suffered greatly since the recent Google update. I have done everything as suggested. I have had all bad links removed over 2 months ago. I have lowered keyword density (not easy since the keyword is in our company name!). I have rewritten various content and bolstered our existing content. What gives? What can I do? As an example the keyword, "maysville plumber" - I rank about 40th for this keyword. The first three pages are filled with websites with literally NO content or no added value. Maysville is a town of about 1k residents - there is no competition. Before the update I was #1 for years on this particular keyword. And this is the case with 35 other cities (mostly small cities, but a few larger ones). Please help me understand or suggest what I can possibly do at this point. We have hundreds of pages of unique content on each and every page. We have zero duplicate content (I have ran tests and crawlers). We have no fishy links. I have not gotten any messages from google on Webmasters. PLEASE HELP!! I asked a similar question a little while back and fixed all of the suggestions. My site is www.akinsplumbing.net.
Intermediate & Advanced SEO | | chuckakins0 -
What is wrong with my site?
I could use suggestions/input. My site is consistently being beaten by doorway pages with zero to no content. In fact, on the keyword "lawrenceville plumber" I am being beaten by a KID with a blank wordpress installation. I really need help determining what my issues are and what I can do to help. According to all of the graders, ranks, etc my site is great. I have written tons of unique content, have added a blog unique articles and self-helps. Please help! My site is www.akinsplumbing.net. Thanks!
Intermediate & Advanced SEO | | chuckakins0 -
Merging three sites to one
Hi guys, I just wanted confirmation if this is the right way to go about doing this. I need to merge three websites and I've never done three websites in to a brand new site before. Ok so we have Sitex.com
Intermediate & Advanced SEO | | Profero
Sitey.com
Sitez.com We've created a SiteB.com SiteB.com has SiteB.com/SiteXCat
SiteB.com/SiteYCat
SiteB.com/SiteZCat Each X,Y and Z have over 1,000 pages. They only have about 10 pages each with Page Authority above 10 and the domains arn't that strong. What i plan to do is: 301 redirect each site domain (X,Y,,Z) to it's corresponding category. e.g. Sitex.com > SiteB.com/SiteXCat 301 redirect each page off X,Y,Z that has a Page Authority above 10 to their new pages on SiteB.com Then, I'm unsure if i should 410 every other URL... I don't think its worht 301 every single URL if they arn't in search results much - but maybe it is if they have a lot of inbound links even with low page authority? Any ideas and does the above seem the best practise? Thanks.0 -
Multiple sites in the same niche
Hi All A question regarding multiple sites in the same niche... If I have say 10 sites all targetting the same niche yet all on different C-class IPs with different hosts, registrars, whois data and ages can I use the same template, or will Google discern a pattern? Basically I have developed a WordPress template which I want to use on the sites albeit with different logos / brand colours. NB/ All of the 10 sites will have unique, original content and they will NOT be interlinked
Intermediate & Advanced SEO | | danielparry1 -
How To Best Close An eCommerce Site?
We're closing down one of our eCommerce sites. What is the best approach to do this? The site has a modest link profile (a young site). It does have a run of site link to the parent site. It also has a couple hundred email subscribers and established accounts. Is there a gradual way to do this? How do I treat the subscribers and account holders? The impact won't be great, but I want to minimize collateral damage as much as possible. Thanks.
Intermediate & Advanced SEO | | AWCthreads0 -
New sites what is the first things to do?
Hi Guys, If you have a baby of a domain which is only a few months old what should be the priorty for getting establish once all the on site stuff has been done? I know the directories are not as important as they use to be but is there a top list that should be worked through steadly to get the new site setup on? Kind regards
Intermediate & Advanced SEO | | ao.com0