Seems like there's a double post here, so I'll copy my answer from the other:

Ok, so a few years ago, I helped a multinational/multidomain corporation with a massive restructuring of their IA and domain portfolio (a total of 21 websites in 15 languages).

You can read all about it here https://moz.com/blog/restructuring-your-website-and-how-to-minimize-traffic-loss

In short, what you wanna do is:

1. Create XML sitemap for old website
2. 301 each individual page from Old Domain to exactly or closest corresponding page on New Domain
3. Ping search engines from the old site with the old XML sitemap, so that they may as quickly as possible discover the move
4. Keep a close eye on Google Webmaster Tools for crawl/error messages and indexation issues
5. Track # of indexed pages on Old and New domain
6. Submit new XML sitemap from New Domain 2 weeks after launch
7. Also consider identifying your most important links via OSE or Majestic, and ask owners to update their links to the new domain.

Hope this helps

Thanks for your input guys! I've almost landed on the following approach:

1. Use this  http://www.chrisains.com/seo-tools/extract-urls-from-web-serps/ to collect a number (3-600) of URLs based on the various problem URL-footprints.
2. Make XML "problem sitemaps" based on above URLs
3. Implement 301s
4. Ping the search engines with the XML "problem sitemaps", so that these may discover changes and see what the site really looks like (ideally reducing the # of indexed pages by about 85%)
5. Track SE traffic as well as index for each URL footprint once a week for 6-8 weeks and follow progress
6. If progress is not satisfactory, then go the URL Profiler route.

Any thoughts before I go ahead?

I would approach this issue with the following:

• I would strongly advice the client against it. This move will hamper the SEO efforts that they're paying for, and they could easily lose 30-60% of their organic traffic. Can they afford this loss?
• If they still insist on doing this, then I think I would keep the old pages as you described. But then that'll give you duplicate issues etc.

I tried thiss and a few others http://www.chrisains.com/seo-tools/extract-urls-from-web-serps/. This gave me about 500-1000 URLs at a time, but included a lot of cut and paste back and forth.

I imagine there must be a much easier way of doing this...

I used Screaming to discover the spider trap (and more), but as far as I know, I cannot use Screaming to import all URLs that Google actually has in its index (or can I?).

A list of URLs actually in Googles index is what I'm after

I started working for this eCommerce site 2 months ago, and my SEO site audit revealed a massive spider trap.

The site should have been 3500-ish pages, but Google has over 30K pages in its index. I'm trying to find a effective way of making a list of all URLs indexed by Google.

Anyone?

(I basically want to build a sitemap with all the indexed spider trap URLs, then set up 301 on those, then ping Google with the "defective" sitemap so they can see what the site really looks like and remove those URLs, shrinking the site back to around 3500 pages)

Thanks for your input guys! I've almost landed on the following approach:

1. Use this  http://www.chrisains.com/seo-tools/extract-urls-from-web-serps/ to collect a number (3-600) of URLs based on the various problem URL-footprints.
2. Make XML "problem sitemaps" based on above URLs
3. Implement 301s
4. Ping the search engines with the XML "problem sitemaps", so that these may discover changes and see what the site really looks like (ideally reducing the # of indexed pages by about 85%)
5. Track SE traffic as well as index for each URL footprint once a week for 6-8 weeks and follow progress
6. If progress is not satisfactory, then go the URL Profiler route.

Any thoughts before I go ahead?

I would approach this issue with the following:

• I would strongly advice the client against it. This move will hamper the SEO efforts that they're paying for, and they could easily lose 30-60% of their organic traffic. Can they afford this loss?
• If they still insist on doing this, then I think I would keep the old pages as you described. But then that'll give you duplicate issues etc.

Seems like there's a double post here, so I'll copy my answer from the other:

Ok, so a few years ago, I helped a multinational/multidomain corporation with a massive restructuring of their IA and domain portfolio (a total of 21 websites in 15 languages).

You can read all about it here https://moz.com/blog/restructuring-your-website-and-how-to-minimize-traffic-loss

In short, what you wanna do is:

1. Create XML sitemap for old website
2. 301 each individual page from Old Domain to exactly or closest corresponding page on New Domain
3. Ping search engines from the old site with the old XML sitemap, so that they may as quickly as possible discover the move
4. Keep a close eye on Google Webmaster Tools for crawl/error messages and indexation issues
5. Track # of indexed pages on Old and New domain
6. Submit new XML sitemap from New Domain 2 weeks after launch
7. Also consider identifying your most important links via OSE or Majestic, and ask owners to update their links to the new domain.

Hope this helps