Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
What tools produce a complete list of all URLs for 301 redirects?
-
I am project managing the rebuild of a major corporate website and need to set up 301 redirects from the old pages to the new ones. The problem is that the old site sits on multiple CMS platforms so there is no way I can get a list of pages from the old CMS. Is there a good tool out there that will crawl through all the sites and produce a nice spreadsheet with all the URLs on it? Somebody mentioned Xenu but I have never used it. Any recommendations? Thanks -Adrian
-
And the winner is... LinkExaminer! Pretty amazing tool, simple to use, but comprehensive - just the thing we need. Thanks for your help Marcin!
Adrian
-
Thanks Marcin and Robert. I will definitely give Xenu Link Sleuth a try first. The Search Engine Optimization Toolkit does look quite awesome though. Thanks for your help. - Adrian
-
There actually many tools which could do the job, both free and paid.
-
Xenu Link Sleuth - free, available at http://home.snafu.de/tilman/xenulink.html
-
Screaming Frog - free (limited)/paid, available at http://www.screamingfrog.co.uk/seo-spider/
-
A1 Website Analyzer - paid, but with a fully functional trial, available at http://www.pendragonexpress.com/hop/a1
-
LinkExaminer - free, available at http://www.analogx.com/contents/download/Network/lnkexam/Freeware.htm
-
Search Engine Optimization Toolkit - free, available at http://www.iis.net/download/seotoolkit
If you just need a crawl, I would recommend 1, 4 or 5. They are both free and work fast. If you want to optimize your content, 5 is a powerhouse.
Hope it helps.
Marcin
-
-
xenu linksleuth is the answer to your question. Good luck. Nothing better I am aware of.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How effective are 301 redirects in passing page rank?
I have a blog which is ranking well for certain terms, and would like to repurpose it to better explain these terms it is ranking for, including updating the url to the new term the blog will be about. The plan being to 301 redirect the old url to new. In the past, I've done this with other pages, and have actually lost much of the rankings that I had earned on the original URL. What is your take on this? Maybe repurpose blog, but maintain original URL just to be on the safe side? Thanks
Technical SEO | | CitimarineMoz0 -
Delete old blog posts after 301 redirects to new pages?
Hi Moz Community, I've recently created several new pages on my site using much of the same copy from blog posts on the same topics (we did this for design flexibility and a few other reasons). The blogs and pages aren't exactly identical, as the new pages have much more content, but I don't think there's a point to having both and I don't want to have duplicate content, so we've used 301 redirects from the old blog posts to the new pages of the same topic. My question is: can I go ahead and delete the old blog posts? (Or would there be any reasons I shouldn't delete them?) I'm guessing with the 301 redirects, all will be well in the world and I can just delete the old posts, but I wanted to triple check to make sure. Thanks so much for your feedback, I really appreciate it!
Technical SEO | | TaraLP1 -
Tools/Software that can crawl all image URLs in a site
Excluding Screaming Frog, what other tools/software to use in order to crawl all image URLs in a site? Because in Screaming Frog, they don't crawl image URLs which are not under the site domain. Example of an image URL outside the client site: http://cdn.shopify.com/images/this-is-just-a-sample.png If the client is: http://www.example.com, Screaming Frog only crawls images under it like, http://www.example.com/images/this-is-just-a-sample.png
Technical SEO | | jayoliverwright0 -
301 redirects delay in picking up
Hi I have been involved in the redesign/development of a website which has up until now had a lot of international traffic. On day of migration I uploaded all the 301 redirects to the website (wordpress) using Simple 301 redirect plugin. I tested a number of them and they appeared to be working. I also submitted the new sitemaps to Search Console. Since migration international traffic - particularly from countries such as india, Phillipines, Sri Lanka etc have significantly dropped off whereas the local traffic and some of the international traffic such as USA has remained fairly consistent. Looking at Analytics and entrances recently it appears as though search results are/were showing a number of pages with 404's (one in particular which received significant traffic and for which I had created a 301 redirection) - I have checked this page using the old url and it re-directs correctly for me and today asked a colleague in India to also check - he is getting the redirection fine. Does Google.in take a significantly longer time to pick these up in search results? Or am I missing something?
Technical SEO | | musthavemarketing0 -
301 Redirect non existant pages
Hi I have 100's of URL's appearing in Search Console for example: ?p=1_1 These go to on to 5_200 etc.. I have tried to do htaccess and the mod rewrite is on as I can redirect directories to the root i.e RewriteRule ^web_example(.*)$ /$1 [R=301,N,L] However I have tried all kinds of variations to redirect ?p= and either it doesn't work at all or it crashes the website. Can anyone point me in the right direction to fix this.
Technical SEO | | Cocoonfxmedia0 -
Tool to Generate All the URLs on a Domain
Hi all, I've been using xml-sitemaps.com for a while to generate a list of all the URLs that exist on a domain. However, this tool only works for websites with under 500 URLs on a domain. The paid tool doesn't offer what we are looking for either. I'm hoping someone can help with a recommendation. We're looking for a tool that can: Crawl, and list, all the indexed URLs on a domain, including .pdf and .doc files (ideally in a .xls or .txt file) Crawl multiple domains with unlimited URLs (we have 5 websites with 500+ URLs on them) Seems pretty simple, but we haven't been able to find something that isn't tailored toward management of a single domain or that can crawl a huge volume of content.
Technical SEO | | timfrick0 -
Sudden drop in Rankings after 301 redirect
Greetings to Moz Community. Couple of months back, I have redirected my old blog to a new URL with 301 redirect because of spammy links pointed to my old blog. I have transfer all the posts manually, changed the permalink structure and 301 redirected every individual URL. All the ranking were boosted within couple of weeks and regained the traffic. After a month I have observed, the links pointed to old site are showing up in Webmaster Tools for the new domain. I was shocked (no previous experience) and again Disavowed all links. Today, all the positions went down for new domain. My questions are: 1. Did the Disavow tool worked this time with new domain? All the links pointed to old domain were devaluated? Is this the reason for ranking drop? Or 2. 301 Old domain with Unnatural links causes the issue? 3. Removing 301 will help to regain few keyword positions? I'm taking this as a case study. Already removed the 301 redirect. Looking for solid discussion.Thanks.
Technical SEO | | praveen4390 -
Mobile URL parameter (Redirection to desktop)
Hello, We have a parallel mobile website and recently we implemented a link pointing to the desktop website. This redirect is happening via a javascript code and results in a url followed by this paramenter: ?m=off Example:
Technical SEO | | echo1
http://www.m.website.com redirects to:
http://www.website.com/?m=off Questions: Will the "http://www.website.com/?m=off" be considered duplicate content with "http://www.website.com" since they both return the same content? Is there any possibility that Google will take into consideration the url ending in "/?m=off"? How should we treat this new url? The webmaster tools URL parameter configuration at the moment isn't experiencing problems but should we submit the parameter anyway in order not to be indexed or should we wait first and see the error response? In case we should submit this for removal... what's the best way to do it? Like this? Parameter: ?m=off Does this parameter change page content seen by the user? - doesn't affect page content Any help is much appreciated.
Thank you!0
