I'm currently using a combination of (mainly free) apps to map my company's website to visualise link flow and indentify isolated content that hasn't been correctly linked in.

I start with a site crawl using Xenu's Link Sleuth from which I can export a list of all pages as well as a list of all the links. I import the two lists into Excel where the data gets cleaned and additional information extracted by a macro.  The final step is to import the link and page data in Cytoscape which performs the visualisation.

Don't be fooled by the biological focus of Cytoscape, it will work with any data as long as it's broken down into nodes (pages) and edges (links). It incorporates a number of visualisation algorithms, and allows you to filter a selection of nodes which can then be copied to a subnetwork and visualised separately.

It handles our small site of 2k pages and 50k internal links with ease, and I know a geneticist who uses it to map networks of over 7k genes with some 300k interactions.

If you're interested, I can provide a copy of the spreadsheet I use for manipulating the raw Xenu data.

1. http://home.snafu.de/tilman/xenulink.html

kJyKm.jpg

He'll need to add [L,R=301] at the end instead of just [L]. IIRC default behaviour is a 302 redirect.

You also can't reference a querystring in the RewriteRule, you have to use RewriteCond.

Hi,

From the .asp in the sample URLs I'm guessing you're hosted on Windows, if that's the case you'll need to get a rewrite mod for IIS such as ISAPI Rewrite 3. We've been using it for about 5 years now and it performs well. Their site has documentation that shows how it works.

You'll need to learn about regex expressions and a tool like Regex Buddy might be helpful.

I'm not aware of an tools that can automate generation, and I think that in your case you're going to need to do some manual work to set it up.

First you'll need a way of linking the old URLs to the new ones. Given the information you've provided, it's not clear how you'll be able to do this, so I'll make an assumption.

Assuming that name-of-the-article is the same as the title of newscontent.asp?id=1133, you'll need to generate a list, in excel for example, that lists the old contentid and the title of that document. You can then use formulae/macros to generate the rewrite rules which you would enter in the .htaccess file.

If you don't have a record of the id = title relationship in your old cms database (assumption!) then you might be able to do it by crawling the old site with a crawling program, exporting the data and then manipulating it. Otherwise you'll have to do it all by hand.

Rewrite rules generally take the form:

RewriteRule oldpageaddress newpageaddress [flags]

You'll also need to use the RewriteCond in order to base the rule on the querystring.

So for your example;

RewriteCond %{QUERY_STRING} ^id=1133$ [NC]

RewriteRule ^newscontent.asp$ ^name-of-the-article$ [L,R=301]

You'd then need to repeat those two statements for each page you want to redirect.

1. .htaccess won't exist on the windows platform unless you installed a rewrite mod on the windows server. If you did then the .htaccesswill be in the root folder of the website (usually) you should check the documentation of the rewrite mod to confirm that.

2. If you have a windows PC then Xenu's Link Sleuth should be able to crawl the old site, you can then extract the information from the files that xenu can export.

3/4. If every unique id needs to get mapped to a unique url then yes, 800 times it is. If you have multiple ids that go to the same page you could do:

RewriteCond %{QUERY_STRING} ^id=113[3-8]$ [NC]

RewriteRule ^newscontent.asp$ ^name-of-the-article$ [L,R=301]

All ids from  1133 to 1138 will now redirect to the same page, you'll have to work out the regexs though.

OK in that case it simplifies things a bit.

In order to do any redirection from id=1136 to unique-article-name you will haveto create the mappings entirely manually.

The two solutions provided are:
Baptiste: On the new linux hosting set up an .htaccess file in the root of the site directory that redirects all id=xxxx requests to a redirect.php file on your server. The redirect.php file will need to interrogate a database with a table of the mappings and automatically redirect to the correct page via php scripting.

Mine: essentially the same as Baptiste's proposal, except that you don't interrogate the database, all the redirections are done using the htaccess file which contains all the mappings.

Either way you will need to manually create the mappings yourself, either in the database or in the htaccess file.

EDIT: Just had a thought, are the page titles of the articles the same between the new site and the old? If they are then you could crawl both sites with Xenu and then use vlookups in excel (or similar) to semi-automatically create your mapping of id = unique-article-name.