Recovering from disaster
-
Short Question: What's the best way to get Google to re-index duplicate URLs?
Long Story:
We have a long ago (1997) established website with a proprietary CMS. Never paid much attention to SEO (other than creating a sitemap) until four months ago. After learning some we started modifying the engine to provide better site to google (proper HTTP codes, consistent URLs to eliminate duplicates - we had something like 15,000 duplicates - etc...)
Things went great for three and half months and we reached the first page on google for our main keyword (very, very competitive keyword). Before the SEO we were getting around 25,000 impressions and 3000 clicks on google. After our SEO efforts, we reached 70,000 daily impressions and more than 7000 daily clicks.
On Aug 30th, 2014, one of our programmers committed a change to the live server by mistake. This small change effectively changed every article's URL by adding either a dash at its end or a dash and a keyword '-test-keyword' (literally).
Nobody noticed anything until two days later as the site worked perfectly for humans. The result of this small code change is that within five days our site practically disappeared from Google's results pages except when one searched for our site's name. Our rank dropped from 8 and 10 to 80 and 100 for our main keywords.
We reverted the change as soon as we noticed the problem, but during those two days, Google's bots went on a binge crawling five times the usual number of page crawled per day.
We've been trying to recover and nothing seems to be working so far. Google's bots aren't crawling the repaired URLs to get the 301 headers back to the original URL and now we still have over 2300 duplicates as reported by the webmaster tools.
Our Google impressions and clicks dropped to way below what we had before we did any SEO, down to 5000 impressions and 1200 clicks (inclusive of our direct domain name search).
During the last 15 days (after we fixed the problem), our duplicate count went from a maximum of 3200, down to 1200, then back up to 2300 without any changes on our end.
we've redone our sitemap and resubmitted it on day 3.
So, what do we do? Do we go through the URLs with 'fetch as Google' function? (that's a bit tedious for 2300 URLs) or we wait for the bots to come around whenever they feel like it? if we do this, should we submit the bad URL, have google fetch it, get the redirect, follow it and then submit the followed URL to the index?
Or is there a better solution that I'm unaware of?
Second question: Is this something to be expected when something like this happens knowing that our inbound link rarely link to the actual articles?
-
Well, after submitting multiple temporary sitemaps and having Google index them, our duplicate counts dropped back to pre-event levels.
However, our rankings haven't improved at all. Actually, if anything, they dropped even further.
At this point it's really starting to look like this is a hit from Panda 4.1 and that we had our URLs change was merely a coincidence. From the looks of it, Google is now marking our site as a low quality site. Now that we know about such a thing, we definitely experienced a 'sinister surge' prior to disaster striking.
Since we've never engaged in any bad behavior on the site and we've always followed google's best practice advice, we're currently at a loss of what could be the reason that we're hit that way. Our content is fresh and high quality (arguably the highest quality in our domain), we have a very decent link profile according to MajesticSEO, so for now, no clue about what's going on really.
Attached is the site's impressions and clicks graph from Webmaster Tools.
-
["knowing that our inbound link rarely link to the actual articles" --> not sure I follow.]
I asked whether it's normal for all ranking to drop even for unaffected pages when pages with no inbound link have issues. For example, our top ranked page for our main keyword didn't change in anyway, not its URL, its description nor its title, yet it's rank tanked after this event.
I like the temporary sitemap idea. Thanks.
-
Once you have all the 301 redirects set up, create a sitemap with all of the old urls and submit that. Google will crawl them and see that they are now 301 redirects and process the data faster. then delete the sitemap.
You should also have a canonical tag on the article pages with the new/current link that should be indexed.
"knowing that our inbound link rarely link to the actual articles" --> not sure I follow.
In general, your rankings should bounce back once google picks up on all of the fixes.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Never recovered from loss of visibility in Google, what did we miss?
Hello, I hope someone can help me, or can recommend someone who could look into this in more detail a a consultancy service or something. Our client https://bit.ly/2IjLD7X experienced a bad drop in visibility early 2017 which we didn't manage to recover from. We had a problem with one main category page that wasn't indexed anymore (it was #1 for a major keyword in the Netherlands). We were able to fix the problem eventually (had to do with the .be being indexed instead due to no hreflang) but never really recovered with that page, only the entire site around that time saw a major drop in visibility. The client went to another agency (a renowned agency in the Netherlands) but after over half a year they haven't been able to recover. Organic visits are still down 45% compared to the same period last year. Another major Dutch keyword they ranked #1 for is now #9 for instance. So basically two SEO agencies (we and the other one) have tried everything we could think of for the past year, but none of us seem to know what the problem is and the owner is quite desperate (it's a small store with only 3 employees and his monthly revenue is only half of what it once was. I'm afraid a quick look into the site won't help anyone find the issue, so if someone has a good idea of how we can help this previous client I would love to hear so!
Technical SEO | | WebdesignTilburg0 -
Recovering from Sitemap Issues with Bing
Hi all, I recently took over SEO efforts for a large e-commerce site (I would prefer not to disclose). About a month ago, I began to notice a significant drop in traffic from Bing and uncovered in Bing Webmaster Tools that three different versions of the sitemap were submitted and Bing was crawling all three. I removed the two out of date sitemaps and re-submitted the up to date version. Since then, I have yet to see Bing traffic rebound and the amount of pages indexed by Bing is still dropping daily. During this time there has been no issue with traffic from Google. Currently I have 1.3 million pages indexed by Google while Bing has dropped to 715K (it was at 755K last week and was on par with Google several months ago). I know that no major changes have been made to the site in the past year so I can't point to anything other than the sitemap issue to explain this. If this is indeed the only issue, how long should I expect to wait for Bing to re-index the pages? In the interim I have been manually submitting important pages that aren't currently in the index. Any insights or suggestions would be very much appreciated!
Technical SEO | | tdawson090 -
Site went down and traffic hasn't recovered
Very curious situation. We have a network of sites. Sunday night one (only one) of our sites goes down, and since then we've seen a loss in traffic across all our sites!! Not only have we seen a loss of traffic, we also saw a loss of indexed pages. A complete drop off from 1.8 million to 1.3 million pages indexed. Does anyone know why one site outtage would affect the rest of them? And the indexed pages? Very confused. Thanks,
Technical SEO | | TMI.com0 -
Recovered from Penalty But.....
Hello,We recovered from Google's manual penalty in January 2014. Afterwards, we changed our site design, fixed our content, disavowed bad links, increased our social presence, tried to engage customers in blog content, etc. But we still couldn't get our domain name back on SERPs. (It wont show our site even on first page if I search "best vpn service" on Google.com)What should we do to bring our domain name back on the SERP?What About Sandbox? What we need to do to get out our domain name from sand box?Any comments or thoughts will be really appreciated 🙂
Technical SEO | | UmairGadit0 -
Redesiging website and how to prevent a SEO disaster?
Hey guys, I'm in the process of re designing my company website. This website runs in Kentico CMS and now I'm moving to drupal CMS. The main reason for the move is that I need a more flexible design with responsiveness and integrate better social aspects. Currently I'm having to deal with a developer whom can't be bothered to update the site. I also find kentico extremely difficult to deal with. The site has good Google rankings and I was thinking about setting up some 301 redirects when the move is completed. Now the question is this site is been live for over 2 years and I have tons of blog posts sitting in here. How can I get these moved? When this is done will they lose their 'age' ? What if I re create the blog posts as it is and then setup 301 re-directs on this? I have G+ authorship and will they be affected by this? Last question is having a responsive site will affect my SEO rankings once moved? I heard the 301 is not considered as a 'safe' thing with Google so what are my options? If anyone can share a link(s) to site upgrades/moving to new design tutorials/best practices articles based on SEO ill be grateful. Thank you very much.
Technical SEO | | Suganthan0 -
SEO Disasters - Links to Stories of Site Moves/Rebuilds Gone Bad?
Hello, everyone.. I am looking for some links to stories, articles, what have you describing medium to large complex sites that have moved, changed CMS, changed domains, etc, and ended up in a total SEO disaster. Really appreciate anything the community here has or can find! Thanks!
Technical SEO | | DuPont0 -
Do I have a chance to recover from penguin?
Hello All, I am an internet retailer www.giftbasket4kids.com hit by Penguin and have made many changes to my site since first being hit on May 2011. It seems I lost my ranking in some keywords on the two following landing pages. I am OK on my other keywords /landing pages. http://www.giftbasket4kids.com/shop-by-category/gift-basket.html keyword “Gift Basket Ideas” http://www.giftbasket4kids.com/pages/birthday-gifts.html Keywords “Birthday gift for kids, boys, Girls, Children” it seems all keyword that includes “Gift, Gifts” and “birthday”. Note: sometime most of the lost keywords come to 2<sup>nd</sup> page for a day or two then they disappear again I received a Google notice first week of August that there are unnatural links pointing to my site. There was couple of paid banner advertising. I stopped the links and request reconsideration. A week later I got response that there are still unnatural link pointing to the site. I request for reconsideration again and asked Google to let me know which links are in question.it is over 3 weeks I did not get answer yet as I suspected. I have optimized most of my product (80 fresh content well optimized product pages) and added a lot new content. I am making links differently. I am using Squidoo lens to promote my products. I do not know what else I can do. I have been told maybe I should restart new site! All my other keywords are ok . I do not want to start over. Can I make new landing pages and promote them? Would it work or the site is penalized for those keywords? I appreciate your advice in advance. If anyone knows of recoveries of a site like mine? I am looking for an SEO who has experience with relates to sites like mine. Do I have a chance to recover from penguin?
Technical SEO | | giftbasket4kids0 -
Panda or Penquin -Website Fell - Shouldn't this Recover?
On March 23rd our site fell 47% in one day. www.TranslationSoftware4u.com but we still held quite a few #1 to #7 rankings on Google and thought it would just recover. Our top keyword "translation software" was #4 , now we are #19 Over the next week I waited to see if it recovered. We have been online 10+ years and always stayed with white hat. I admit to learning as I go over the years but always felt content was king so I focused on information. I really do not see my site as using spam techniques but maybe I am missing something on the way I have it. March 23rd, major drop -47% On April 2nd I started with SEO MOZ and the Research tools showed we had duplicate content warning. This was from a blog we were trying to start that only had 7 posts but it had about 20 tags per post. I did not realize that tags actually created that post under that tag. I went in and deleted the tags again being stupid and not realizing it was then making that come up 404. The blog was so small we do not get hits on it anyway so hoping it just clears itself up. ( still get duplicate warning on our directory due to using "php Link Directory", but it's due to how it reuses the title tag and description, 2 instances per category page"). Still trying to fix the php directory issue. Seems many others are running it and did not have a drop. April 24th, we dropped another -10% It keeps falling -70% now. I have gone through the site and tried to clean up any warnings like duplicate title tags, meta descriptions. With regards to links I put up a small web directory with some reciprocal linking. Our product translates languages but software is not the same as a human so we often set clients up with human translators, the directory is a nice place to help our customers find a translator or see online tools that can help. The links were not excessive, there were maybe 100 links. After the fall I went in and found some translators had gone out of business so I deleted those, I am down to 65 links now, about 45 are exchanges. I have submitted to some online directories manually, but looking back through the links there is not really anything that makes me concerned. The link back to my site was really the most neglected SEO thing I did. Again concentrating on content. I did find a few links that I was not happy about but I did not put those links so had no control. I have been working on cleaning up my title tags, and making sure the content just reads better. I have been hoping that my site would just start recovering but it keeps sliding. Has anyone seen recovery from the updates. Should I see anything yet? I cannot seem to get Google to return to the site and reindex. Am I doing somethign spammy on my site and I do not realize it? Thanks for any advice in advance!
Technical SEO | | Force70