Recovering from disaster
-
Short Question: What's the best way to get Google to re-index duplicate URLs?
Long Story:
We have a long ago (1997) established website with a proprietary CMS. Never paid much attention to SEO (other than creating a sitemap) until four months ago. After learning some we started modifying the engine to provide better site to google (proper HTTP codes, consistent URLs to eliminate duplicates - we had something like 15,000 duplicates - etc...)
Things went great for three and half months and we reached the first page on google for our main keyword (very, very competitive keyword). Before the SEO we were getting around 25,000 impressions and 3000 clicks on google. After our SEO efforts, we reached 70,000 daily impressions and more than 7000 daily clicks.
On Aug 30th, 2014, one of our programmers committed a change to the live server by mistake. This small change effectively changed every article's URL by adding either a dash at its end or a dash and a keyword '-test-keyword' (literally).
Nobody noticed anything until two days later as the site worked perfectly for humans. The result of this small code change is that within five days our site practically disappeared from Google's results pages except when one searched for our site's name. Our rank dropped from 8 and 10 to 80 and 100 for our main keywords.
We reverted the change as soon as we noticed the problem, but during those two days, Google's bots went on a binge crawling five times the usual number of page crawled per day.
We've been trying to recover and nothing seems to be working so far. Google's bots aren't crawling the repaired URLs to get the 301 headers back to the original URL and now we still have over 2300 duplicates as reported by the webmaster tools.
Our Google impressions and clicks dropped to way below what we had before we did any SEO, down to 5000 impressions and 1200 clicks (inclusive of our direct domain name search).
During the last 15 days (after we fixed the problem), our duplicate count went from a maximum of 3200, down to 1200, then back up to 2300 without any changes on our end.
we've redone our sitemap and resubmitted it on day 3.
So, what do we do? Do we go through the URLs with 'fetch as Google' function? (that's a bit tedious for 2300 URLs) or we wait for the bots to come around whenever they feel like it? if we do this, should we submit the bad URL, have google fetch it, get the redirect, follow it and then submit the followed URL to the index?
Or is there a better solution that I'm unaware of?
Second question: Is this something to be expected when something like this happens knowing that our inbound link rarely link to the actual articles?
-
Well, after submitting multiple temporary sitemaps and having Google index them, our duplicate counts dropped back to pre-event levels.
However, our rankings haven't improved at all. Actually, if anything, they dropped even further.
At this point it's really starting to look like this is a hit from Panda 4.1 and that we had our URLs change was merely a coincidence. From the looks of it, Google is now marking our site as a low quality site. Now that we know about such a thing, we definitely experienced a 'sinister surge' prior to disaster striking.
Since we've never engaged in any bad behavior on the site and we've always followed google's best practice advice, we're currently at a loss of what could be the reason that we're hit that way. Our content is fresh and high quality (arguably the highest quality in our domain), we have a very decent link profile according to MajesticSEO, so for now, no clue about what's going on really.
Attached is the site's impressions and clicks graph from Webmaster Tools.
-
["knowing that our inbound link rarely link to the actual articles" --> not sure I follow.]
I asked whether it's normal for all ranking to drop even for unaffected pages when pages with no inbound link have issues. For example, our top ranked page for our main keyword didn't change in anyway, not its URL, its description nor its title, yet it's rank tanked after this event.
I like the temporary sitemap idea. Thanks.
-
Once you have all the 301 redirects set up, create a sitemap with all of the old urls and submit that. Google will crawl them and see that they are now 301 redirects and process the data faster. then delete the sitemap.
You should also have a canonical tag on the article pages with the new/current link that should be indexed.
"knowing that our inbound link rarely link to the actual articles" --> not sure I follow.
In general, your rankings should bounce back once google picks up on all of the fixes.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Recovering from a Google penalty
Hi there, So about 3.5 weeks ago I noticed my website (www.authenticstyle.co.uk) had gone from ranking in second place for our main key phrase "web design dorset" to totally dropping off the SERP's for that particular search phrase - it's literally no where to be seen. It seems that other pages of my website still rank, but the homepage. I then noticed that I had an unread alert in my Google Search Console account to say that a staging site we were hosting on a subdomain (the subdomain was domvs.authenticstyle.co.uk) had hacked content - it was a couple of PDF files with weird file names. The strange thing is we'd taken this staging site down a few weeks earlier, BUT one of my staff had left an A record set up in our Cloudflare account pointing to that staging server - they'd forgotten to remove it when removing the staging site. I then removed the A record, myself and submitted a reconsideration request on Google Search Console (which I still haven't received confirmation of) in the hope of everything sorting itself out. Since then I've also grabbed a Moz Pro account to try and dig a little deeper, but without any success. We have a few warnings for old 404's, some missing meta descs on some pages, and some backlinks that have accumulated over time that have hghish spam rating, but nothing major - nothing that would warrant a penalty as far as I can tell. From what I can make out, we've been issued a penalty on our homepage only, but I don't understand why we would get penalised for hacked content if that site domvs.authenticstyle.co.uk no longer existed (would it just be due to that erroneous A record we forgot to remove?). I contacted a few freelance SEO experts and one came back to me saying I'd done everything correctly and that I should see our site appearing again in a few days after submitting the reconsideration request. Its been 3 weeks and nothing. I'm at a huge loss as to how my site can recover from this. What would you recommend? I even tried getting our homepage to rank for a variation of "web design dorset", but it seems our homepage has been penalised for anything with "dorset" in the keyphrase. Any pointers would be HUGELY appreciated. Thanks in advance! Will
Technical SEO | | wsmith7270 -
Never recovered from loss of visibility in Google, what did we miss?
Hello, I hope someone can help me, or can recommend someone who could look into this in more detail a a consultancy service or something. Our client https://bit.ly/2IjLD7X experienced a bad drop in visibility early 2017 which we didn't manage to recover from. We had a problem with one main category page that wasn't indexed anymore (it was #1 for a major keyword in the Netherlands). We were able to fix the problem eventually (had to do with the .be being indexed instead due to no hreflang) but never really recovered with that page, only the entire site around that time saw a major drop in visibility. The client went to another agency (a renowned agency in the Netherlands) but after over half a year they haven't been able to recover. Organic visits are still down 45% compared to the same period last year. Another major Dutch keyword they ranked #1 for is now #9 for instance. So basically two SEO agencies (we and the other one) have tried everything we could think of for the past year, but none of us seem to know what the problem is and the owner is quite desperate (it's a small store with only 3 employees and his monthly revenue is only half of what it once was. I'm afraid a quick look into the site won't help anyone find the issue, so if someone has a good idea of how we can help this previous client I would love to hear so!
Technical SEO | | WebdesignTilburg0 -
Recovered from Penalty But.....
Hello,We recovered from Google's manual penalty in January 2014. Afterwards, we changed our site design, fixed our content, disavowed bad links, increased our social presence, tried to engage customers in blog content, etc. But we still couldn't get our domain name back on SERPs. (It wont show our site even on first page if I search "best vpn service" on Google.com)What should we do to bring our domain name back on the SERP?What About Sandbox? What we need to do to get out our domain name from sand box?Any comments or thoughts will be really appreciated 🙂
Technical SEO | | UmairGadit0 -
Recovering from Blocked Pages Debaucle
Hi, per this thread: http://www.seomoz.org/q/800-000-pages-blocked-by-robots We had a huge number of pages blocked by robots.txt by some dynamic file that must have integrated with our CMS somehow. In just a few weeks hundreds of thousands of pages were "blocked." This number is now going down, but instead of by the hundreds of thousands, it is going down by the hundreds and very sloooooowwwwllly. So, we really need to speed up this process. We have our sitemap we will re-submit, but I have a few questions related to it: Previously the sitemap had the <lastmod>tag set to the original date of the page. So, all of these pages have been changed since then. Any harm in doing a mass change of the <lastmod>field? It would be an accurate reflection, but I don't want it to be caught by some spam catcher. The easy thing to do would be to just set that date to now, but then they would all have the same date. Any other tips on how to get these pages "unblocked" faster? Thanks! Craig</lastmod></lastmod>
Technical SEO | | TheCraig0 -
SEO Disasters - Links to Stories of Site Moves/Rebuilds Gone Bad?
Hello, everyone.. I am looking for some links to stories, articles, what have you describing medium to large complex sites that have moved, changed CMS, changed domains, etc, and ended up in a total SEO disaster. Really appreciate anything the community here has or can find! Thanks!
Technical SEO | | DuPont0 -
A huge drop in rankings since last 10 days, and not recovered yet.
Hi Mozzers, I have a serious topic to discuss and want help from the experts here. Our website has 6 PR and we have been consistency staying at the top for very competitive terms in the niche. Since last Friday (24th February, 2012) we have been facing massive fluctuation in the rankings for most of the keywords we are focusing on. After this fall, we checked the following details but didn’t find any serious/critical issue that might be contributing towards these fluctuations:- We analyzed Google webmaster tools, there’s no update/warning from Google regarding any negative activity and other things seem to be normal. We checked our website through site search (site: www.domain.com) and found that we haven’t lost any indexed pages and things appear normally as they used to. So, we are sure that we haven’t been banned or penalized. We also cross verified our link building and other promotional activities and we didn’t find anything suspicious that could lead to such a big fluctuation. The drop is really big, some keywords went to 5th or 6th page from top 3 position; some keywords are not in top 200 or 300 spots which were usually staying put between 5th to 10th position. We have analyzed a lot but haven’t come to know the reason why we are facing this fluctuation. Our website is 4 years old and this kind of fluctuation has happened for the first time. Has anyone faced this kind of issue before? I’m looking forward to your support in identifying this trouble. Thanks
Technical SEO | | ValSmith0 -
Redesign an SEO-Disaster | Help with Redirects of Gray Hat Pages
Hi gang. I'm a new SEO and I'm currently working on the redesign of a website. I have just discovered a ton of hidden pages that are filled with duplicate content, basically reiterating the main keyword in a variety of different variations. Each page is titled with the variation on the keyword phrase and then has one paragraph of text very similar to the previous page, etc. Here is an example of one of the offensive pages (nice lookin' site, eh?): http://www.vasectomy-reversals.com/vasectomy_reversal_surgery.html The new site will not have any of these pages. I'm writing the 301 redirects now and want to redirect these offensive pages to the most relevant page on the new site. But, I'm afraid to redirect the offensive pages. Should I leave them alone, or can I have the former developer remove them? Help. Don't know how to handle these pages and their redirects. Thanks for your help! ~ Mills
Technical SEO | | Mills0 -
How to recover after blocking all the search engine spiders?
I have the following problem - one of my clients (a Danish home improvement company) decided to block all the international traffic (leaving only Scandiavian one), because they were getting a lot of spammers using their mail form to send e-mails. As you can guess this lead to blocking Google also since the servers of Google Denmark are located in the US. This lead to drop in their rankings. So my question is - What Shall I do now - wait or contact Google? Any help will be appreciated, because to be honest I had never see such thing in action until now 😄 Best Regards
Technical SEO | | GroupM0