Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How to prevent development website subdomain from being indexed?
-
Hello awesome MOZ Community!
Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.
An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.
The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.
My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?
Thanks!
-
Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.
You can find information on setting this up here.
-
Thanks for your prompt feedback Oleg!
Your proposed action is exactly what we're implementing.
Randy Holland, Sprout Digital
-
So....
- If the dev site has not been indexed yet, you can block crawlers via robots.txt
- If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again
As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does changing content and design of the website gonna affect my all the backlinks i have made till now
i have been working on my link profile for a month now, after learning about 5 step moz methodology i have decided that i would like to change all of the content of my site and taylor it to what my customers need, am i gonna loose all the domain authority if make changes? if it gonna affect, hows that gonna come out
Web Design | | calvinkj0 -
Website Redesign - What to do with old 301 URLs?
My current site is on wordpress. We are currently designing a new wordpress site, with the same URLs. Our current approach is to go into the server, delete the current website files and ad the new website files. My current site has old urls which are 301 redirected to current urls. Here is my question. In the current redesign process, do i need to create pages for old the 301 redirected urls so that we do not lose them in the launch of the new site? or is the 301 command currently existing outside of our server so this does not matter? Thank you in advance.
Web Design | | CamiloSC0 -
Website organic traffic unchanged, impressions took a 98% drop in the last week.
Hi all, I have a very curious predicament and I'd be grateful if someone could shed some light on the situation. As mentioned in the title, organic traffic to our website has remained unchanged, but organic impressions have taken a 98% drop in the last week. This happened suddenly over one day; on October 22, impressions were 700, on October 23, they were 500, and on October 24 they drastically dropped to 50. The next two days they were at 22 and then up to 35. Organic traffic, however, showed the normal "weekend drop" as of October 24, and is still showing normal level (even increased a bit) continuing into this week. These are organic impressions according to Google Analytics and Google Webmaster tools. We did perform a complete site redesign a month ago. Could this be an effect from the redesign? We also noticed drop in Domain Authority, but our competitors suffered a similar (if not greater) drop as well, so we wondered if it could be due in part to the algorithm update. If anyone could shed some light on the situation I would be so appreciative! Thanks!
Web Design | | Joanne_Pendon0 -
Lots of Listing Pages with Thin Content on Real Estate Web Site-Best to Set them to No-Index?
Greetings Moz Community: As a commercial real estate broker in Manhattan I run a web site with over 600 pages. Basically the pages are organized in the following categories: 1. Neighborhoods (Example:http://www.nyc-officespace-leader.com/neighborhoods/midtown-manhattan) 25 PAGES Low bounce rate 2. Types of Space (Example:http://www.nyc-officespace-leader.com/commercial-space/loft-space)
Web Design | | Kingalan1
15 PAGES Low bounce rate. 3. Blog (Example:http://www.nyc-officespace-leader.com/blog/how-long-does-leasing-process-take
30 PAGES Medium/high bounce rate 4. Services (Example:http://www.nyc-officespace-leader.com/brokerage-services/relocate-to-new-office-space) High bounce rate
3 PAGES 5. About Us (Example:http://www.nyc-officespace-leader.com/about-us/what-we-do
4 PAGES High bounce rate 6. Listings (Example:http://www.nyc-officespace-leader.com/listings/305-fifth-avenue-office-suite-1340sf)
300 PAGES High bounce rate (65%), thin content 7. Buildings (Example:http://www.nyc-officespace-leader.com/928-broadway
300 PAGES Very high bounce rate (exceeding 75%) Most of the listing pages do not have more than 100 words. My SEO firm is advising me to set them "No-Index, Follow". They believe the thin content could be hurting me. Is this an acceptable strategy? I am concerned that when Google detects 300 pages set to "No-Follow" they could interpret this as the site seeking to hide something and penalize us. Also, the building pages have a low click thru rate. Would it make sense to set them to "No-Follow" as well? Basically, would it increase authority in Google's eyes if we set pages that have thin content and/or low click thru rates to "No-Follow"? Any harm in doing this for about half the pages on the site? I might add that while I don't suffer from any manual penalty volume has gone down substantially in the last month. We upgraded the site in early June and somehow 175 pages were submitted to Google that should not have been indexed. A removal request has been made for those pages. Prior to that we were hit by Panda in April 2012 with search volume dropping from about 7,000 per month to 3,000 per month. Volume had increased back to 4,500 by April this year only to start tanking again. It was down to 3,600 in June. About 30 toxic links were removed in late April and a disavow file was submitted with Google in late April for removal of links from 80 toxic domains. Thanks in advance for your responses!! Alan0 -
3 Brands, 3 Services, 3 Different Websites Right?
My client was told that having 1 website for 3 different brands/services is better than having 3 websites. I need your help to prove my value to a new client. This client has worked with Reach Local on PPC for some time and when they first got started the Reach Local Markering Consultant told this cleint that they needed to have one site for better SEO purposes. The client was told that Google ranks websites higher if they have more paid traffic going to them. I've been doing this for long enough to realize this does not help ranking, at least not enough to make a difference. Keep in mind this is for 3 different companies. One company does plumbing, another electrical and the last one does air conditioning. They also have 4 locations but only two locations have mutliple services opperating out of them. I understand these 2 location will not have there own Google+ Local / Places listing. Using the same address for 2 different business and expecting a first page ranking is just not possible. Right now when you visit the clients website you see a logo that rotates with a banner section that follows the logo rotation. First you see the AC Company and then the Plumbing etc. I see this as confusing to the end user and it is more work to get it ranked for SEO. I recommended that we build 3 speerate websites for each service and just list out all the addresses that the company services on the contact page. I would also design inside the footer links to the other services for branding purposes. Please share your thoughts on how you would handle this if you were doing the SEO for your own 3 different business services. I really appreicate any input/insight to this. Thank you so much in advance!!!!
Web Design | | 1SMG0 -
Need to rebuild client's flash website
I am working with their web designer and need to figure out a way to rebuild their site which is currently all in flash. I was wondering if there was a way to do this without spending a ton of time in completely re-doing the site from scratch.
Web Design | | awalker840 -
Google also indexed trailing slash version - PLEASE HELP
Hi Guys, We redesigned the website and somehow our canonical extension decided to add a trailing slash to all URLs. Previously our canonical URLs didn't have a trailing slash. During the redesign we haven't changed the URLs. They remained same but we have now two versions indexed. One with trailing slash one without. I've now fixed the issue and removed the the trailing slash from canonical URLs. Is this the correct way of fixing it? Will our rankings be effected in a negative way? Is there anything else I need to do. The website went live last Tuesday. Thanks
Web Design | | Jvalops0 -
Facebook code being duplicated? (Any developers mind taking a peek?)
I'm using a few different plug ins to give me various Facebook functions on my site. I'm curious there are any developers out there would could take a look at my source code and see if it looks there is some code being duplicated that's slowing down my site. Thanks so much!
Web Design | | NoahsDad0