Should /dev folder be blocked?
-
I have been experiencing a ranking drop every two months, so I came upon a new theory this morning...
Does Google do a deep crawl of your site say every 60-90 days and would they penalize a site if they crawled into your /dev area which would contain pretty the exact same urls and content as your production environment and therefore penalize you for duplicate content?
The only issue I see with this theory is that I have been penalized only for specific keywords on specific pages, not necessarily across the board.
Thoughts?
What would be the best way to block out your /dev area?
-
Hey Ryan,
It isn't really a competition thing because I bounce back to the same or better sport after about 30 days, so it seems very algorithmic. I just haven't been able to figure what/how I am getting penalized. However, this morning while looking at some rankings I noticed a dev page was indexed and in Google's rankings so I said "Ah oh...." and went to see my developer hadn't blocked out our development directory... which means if Google deep indexes a site every so often they would probably crawl it and find an exact copy of the live site...
The 60-90 deep indexing would also make sense as the ranking drop has always occurred around the 27th of the month.
Once I block out /dev what do you think would be the best way to get Google to re-crawl the site and perhaps remove any duplicate issues - delete out my robots.txt and sitemaps from Google and re-submit them?
-
It is a great idea to block the development portion of your site. You can do that with your robots.txt file:
User-Agent: *
Disallow: /dev
But you're right, it doesn't sound like the culprit in your case. Its more likely that your competition is slowly gaining the edge on you. Make sure your on-page SEO is optimized and then try to get some more inbound links to your pages.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Magento 2.1 Multi Store / SEO
This is quite technical but I'm hoping a Magento expert can clear this up for me. Currently my company has two websites on separate Opencart platforms. What I'm doing now is building a Magento website and using the multi store function as well as a few modules to combine the two sites, the aim being that the link juice is shared and I can focus my SEO efforts on the one site instead of two, thus reducing my workload while maintaining the benefits. This is the intended layout: www.domain.com www.domain.com/us I have created a sub-folder (not a subdomain) as this seems to be the best way to share link juice between the new, combined sites (as well as 301s from the old, redundant site). At the moment I have created 2 separate websites, stores and store views (see attached) and have configured it according to the Magento guide, so I know that technically this is correct but I need to make sure that I have done it correctly in relation to SEO. Is the sub-folder set up correctly for instance? Currently the only files to populate that sub-folder are a htaccess, error log and index.php (see attached). Also, is there anything I could be missing in relation to SEO within the parameters of what I am trying to achieve? Additionally, only one store view appears in the "change store view" section of the home page. This is causing me to question if I have set it up correctly, because I had assumed both store views would appear even if they were under different websites (attached). OR do I simply use the same website and create two stores and store views? Do I also need to create a separate database for each website/store/store view? I would very much appreciate if someone could help out here. Thank you. In1Gi7t pyfM03y nUQoMz1
Web Design | | moon-boots0 -
Anybody with a business site using Webflow CMS or another non-WordPress CMS/ sitebuilder?
If you have built your business site using Webflow CMS, kindly share your insights on how good their CMS is for SEO and how easy it is to rank a site. If you are not familiar with Webflow but have had a positive experience with another non-WordPress CMS/ sitebuilder, kindly share your opinions.
Web Design | | Blu_Ocean99.0 -
Privacy Policy: index it/? And where to place it?
Hi Everyone, Two questions, first: should you allow google to index your privacy policy? Second: for a service based site (not e-commerce, not selling anything) should you put the policy in the footer so it's site wide or just on the "contact us" form page? Best, Ruben
Web Design | | KempRugeLawGroup0 -
Google tag manager on blocked beta site - will it phone home to Google and cause site to get indexed?
We want to develop a beta site, in a directory with the robots.txt blocking bots. We want to include the Google Tag Manager tags and event layer tracking code on this beta site. My question is that by including the Google Tag Manager code, that phones home to Google, will it cause Google to index this beta site when we don't want it indexed?
Web Design | | CFSSEO0 -
Image with 100% width/height - bad ranking?
Hi, we have some articles like this: http://www.schicksal.com/Orakel/Freitag-13 The main image has a width of 100% and a height of 100%. Today, I've discovered that GWT Instant Preview has some troubles with rendering the page. We have CSS rules to deliver the image with the right dimensions. If a bot like google is not sending any screen height / width we assume the screen size is 2560x1440. Does this harm the ranking of the page? (Content starts below the fold/image) What is a "default" screen size for google? How do they determine if something is "above the fold"? Any tips or ideas? Best wishes, Georg.
Web Design | | GeorgFranz0 -
White Text / Black Background & SEO Impact
Does anyone know of any testing / studies with evidence that Google prefers dark text on a light background vs. light text on a dark background? I have a website that currently has light text on a black background, and really like the way it looks, but am concerned that the style may be hurting SEO. Moreover, redesigning something inverse with the same quality would be a large project and fairly costly, so I'd like to make sure the benefit will really be worth the cost before moving forward.
Web Design | | Bromtec0 -
Redirects (301/302) versus errors (404)
I am not able to convincingly decide between using redirects versus using 404 errors. People are giving varied opinions. Here are my cases 1. Coding errors - we put out a bad link a. Some people are saying redirect to home page; the user at least has something to do PLUS more importantly it does NOT hurt your SEO ranking. b. Counter - the page ain't there. Return 404 2. Product removed - link1 to product 1 was out there. We removed product1; so link1 is also gone. It is either lying in people's bookmarks, OR because of coding errors we left it hanging out at some places on our site.
Web Design | | proptiger0 -
What site do you admire/like for its SEO - technical, content, whatever - and why?
I am gathering examples of great SEO'd sites and would appreciate your examples. The rationale can be anything - great SEO structure, great linking, solid content - you think stands out. Thank you!
Web Design | | josh-riley0