Quickest way to deindex a large number of pages
-
Our site was recently hacked by spammers posting fake content and bringing down our servers, etc. After a few months, we finally figured out what was going on and fixed the issue. However, it turns out that Google has indexed 26K+ spammy pages and we've lost page rank and search engine rankings as a result.
What is the best and fastest way to get these pages out of Google's index?
-
Given that I'm sure you've removed these pages from your site, there will be no page to which to add a meta-noindex tag.
Disallowing these pages in robots.txt in no way signals to the search engines that they should be removed from the index, just that they should no longer be crawled. Given that they're already indexed, blocking in robots.txt would potentially save some "crawl budget" but wouldn't do anything to remove them from the index.
So submitting them to the URL Removal Tool would be by far the most effective, along with an explanation.
You'll also want to keep a very close watch on your penalty warnings within Webmaster Tools. If you get flagged, you'll want a complete history of the issue and the steps you've taken to address it in order to prepare a reinclusion request.
Lastly, don't forget to submit these same URLs to the Bing Webmaster Tools Block URLs tool. You may not get a massive amount of traffic from Bing, but there's no sense throwing it away, since you've already prepared the URL removal list anyway.
Hope that helps?
Paul
-
Yup. Just wanted to add as well that if these pages are in a particular directory, then you can deindex the entire directory in one command using the URL removal tool.
-
Disallow in robots.txt
Add a noindex meta tag to these pages
Request Google to remove the URLs from their index via WMT URL removal request
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it Okay to have "No Response" pages?
Hi all, I can see some "No Response" pages which gives a error message "Site cannot be reached" or keeps on loading but don't. I have got this list from Screaming from spider tool. Do we need to fix these or ignore? Thanks
Algorithm Updates | | vtmoz0 -
Do we need to maintain consistency in page titles suffix?
Hi all, We usually give "brand & primary keyword" across all pages in website like "vertigo tiles". Do we need to maintain this suffix across all page titles? What if we change according to the page? Will Google downlook for not maintaining these page titles suffix like I mentioned? Thanks
Algorithm Updates | | vtmoz0 -
How I can check if Google and other search engines will properly cache a page (a dynamic one)?
My site is currently disallowing search engine bots with the help of robots.txt. These dynamic pages can be crawled using Screamingfrog since they are linked to a static category page which is also linked to the homepage. Thanks in advance!
Algorithm Updates | | esiow20130 -
A Serious drop in Pages crawled per day
On 21st April ,I spotted a sudden decrease in pages crawled per day.Previously it was about 5,000 bust after the drop it reached to 225.From the crawl rate never spiked. Here is my website url - http://www.wpstuffs.com/ 8fQHW2G.png
Algorithm Updates | | vividvilla0 -
Bing's indexed pages vs pages appearing in results
Hi all We're trying to increase our efforts in ranking for our keywords on Bing, and I'm discovering a few unexpected challenges. Namely, Bing is reporting 16000+ pages have been crawled... yet a site:mywebsite.com search on Bing shows less than 1000 results. I'm aware that Duane Forrester has said they don't want to show everything, only the best. If that's the case, what factors must we consider most to encourage Bing's engine to display most if not all of the pages the crawl on my site? I have a few ideas of what may be turning Bing off so to speak (some duplicate content issues, 301 redirects due to URL structure updates), but if there's something in particular we should monitor and/or check, please let us know. We'd like to prioritize 🙂 Thanks!
Algorithm Updates | | brandonRT0 -
Canonicalization on more than one page?
is it proper to "canocalize" more than one page in a site? Or should it only be on the home page? eg: http://www.sundayschoolnetwork.com">
Algorithm Updates | | sakeith0 -
Privacy page ranking above home page in serps
I'm using OSE to try and get some clues as to why my privacy page would rank higher than my home page. Could anyone help me figure out which metrics to review to rectify the issue? My key word is: Mardi Gras Parade Tickets The url that is ranking is <cite>www.mardigrasparadetickets.com/pages/privacy</cite> I'm happy to be ranking in the top 3 for the keyword, but I'd rather hoped it wouldn't be my privacy page. Any help would be awesome, Cy
Algorithm Updates | | Nola5040 -
Is it OK to 301 redirect the index page to a search engine friendly url
Is it OK to 301 redirect the index page to a search engine friendly url.
Algorithm Updates | | WinningInch0