Quickest way to deindex large parts of a website
-
Hey there,
my clients website was set up with subdirectories for almost every country in the world plus multiple languages in each country. The content in each subfolder is (almost) identical. So no surprise: They have a big problem with duplicate content and ranking fluctuations.
Since they don't want to change the site's structure I recommended limiting the languages available in each subfolder with robots.txt. However before doing this we marked the contents to be exluded with noindex, nofollow. It's only been 2 days now but I hardly notice any decline in the number of indexed pages.
I was therefore wondering if it would speed up things if I marked the pages with just noindex instead of noindex and nofollow.
It would be great if you could share your thoughts on that.
Cheers,
Jochen
Hey there,
my clients website was set up with subdirectories for almost every country in the world plus multiple languages in each country. The content in each subfolder is (almost) identical. So no surprise: They have a big problem with duplicate content and ranking fluctuations.
Since they don't want to change the site's structure I recommended limiting the languages available in each subfolder with robots.txt. However before doing this we marked the contents to be exluded wiht noindex, nofollow. It's only been 2 days now but I hardly notice any decline in the number of indexed pages.
I was therefore wondering if it would speed up things if I marked the pages with just noindex instead of noindex and nofollow.
It would be great if you could share your thoughts on that.
Cheers,Jochen -
Thanks for the hint Dirk! I've used the tool and it works great. I even found a handy chrome extension ("WebMaster Tools - Bulk URL removal") that made the removal of my 3,000 subdirectories very smooth and saved me about 25 hours of manual work!
WebMaster Tools - Bulk URL removal
-
Hi,
There was a similar question a few days ago: https://moz.com/community/q/is-there-a-limit-to-how-many-urls-you-can-put-in-a-robots-txt-file
Quote: Google Webmaster Tools has a great tool for this. If you go into WMT and select "Google index", then "remove URLs". You can use regex to remove a large batch of URLs then block them in robots.txt to make sure they stay out of the index.
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Different ways to implement schema markup?
Hey guys, What are all the ways you can implement schema markup Plugins (e.g. wordpress, shopify) Google data highlighter in GSC Google Structured Data Markup Helper Google Tag Manager Also is google data highlighter different to Google Structured Data Markup Helper? Cheers.
Intermediate & Advanced SEO | | michel_80 -
Website Traffic Is Down
Hi, My Website www.financeninvestments.com is down for almost now 2 years. I was receiving the good traffic before this but now the traffic is almost down. I want to again do something to get my Traffic back with some consistent efforts. So what efforts should i do to make this back.Pls suggest.
Intermediate & Advanced SEO | | rahulsoni250 -
Check website update frequency?
Is the tools out there that can check our frequently website is updated with new content products? I'm trying to do an SEO analysis between two websites. Thanks in advance Richard
Intermediate & Advanced SEO | | seoman100 -
D.A. and Link Juice from certain websites
Hello, i'm a bit confused concerning some link buildings. What happens with backlinks from blogs coming from powerful domains such as abc.yahoo.com or abc**.over-blog.com or abc.blogspot.com?** Meaning, everyone that creates a blog over there will have a PA of 0/100 but a DA of 80 or 90/100.
Intermediate & Advanced SEO | | prozis
Will Google consider DA on these cases? I'm confused because it can't be that simple that someone creates a website, and after some months they will have like 10 PA but still the 90/100 DA and their links can be a powerful backlink. Can you explain me how Google sees that? So, if I have a link coming from a blog on those domains will it be better than any other with same PA but lower DA?0 -
Training Website Improvements...
Hi Folks, I'm in the process of going over our corporate website with a view to improving on-page optimisation, layout, design and user experience and I would like your feedback on what you think I should improve or change with respect to SEO. Some of my ideas include: Restructure Home Page to Better Show Our Services Possibly Add a Slider to the Home Page (I know engagement rates with these are generally low) Restructure the Course Pages Completely (https://purplegriffon.com/courses/itil-training/itil-foundation-training/itil-foundation) Restructure the Events Pages Completely (https://purplegriffon.com/event/2028/itil-foundation) Improve & Streamline the Booking Process AJAXIFY the Booking Process Improve Responsive Elements I'm also interested in conducting user testing before I go ahead and make any changes. What are your thoughts? What would you change? Thanks. Gaz
Intermediate & Advanced SEO | | PurpleGriffon0 -
Questions about websites coupon codes
Hi guys, i have 2 questions about my website of coupon codes: should i do redirect people of google to mywebsite, e.g. Someone is looking coupons for Sony or LG and arrive to brand Sony in my website but i show him offers for Sony in Amazon when he click in some offer, ¿that is correct? Footer images links. I saw many sites that put their logos in footer of online stores to get authority, ¿should i do that? Thank you so much.
Intermediate & Advanced SEO | | pompero990 -
Is it Penguin or is it just a shoddy website?!
Right, I'm struggling and could really do with a different perspective on a website I am working on http://www.data-contracts.co.uk/ This is an Analytics image for March through to July of organic unique visitors http://imgur.com/Y2zaekP The page for concrete repair is struggling in the rankings and is currently residing somewhere like page 15 on Goooooogle. Is there anyway back??? Y2zaekP
Intermediate & Advanced SEO | | Hughescov0 -
Moving Content To Another Website With No Redirect?
I've got a website that has lots of valuable content and tools but it's been hit too hard by both Panda and Penguin. I came to the conclusion that I'd be better off with a new website as this one is going to hell no matter how much time and money I put in it. Had I started a new website the first time it got hit by Penguin, I'd be profitable today. I'd like to move some of that content to this other domain but I don't want to do 301 redirects as I don't want to pass bad link juice. I know I'll lose all links and visitors to the original website but I don't care. My only concern is duplicate content. I was thinking of setting the pages to noindex on the original website and wait until they don't appear in Google's index. Then I'd move them over to the new domain to be indexed again. Do you see any problem with this? Should I rewrite everything instead? I hate spinning content...!
Intermediate & Advanced SEO | | sbrault741