Broken Inner Links - Tool Recommendations?
-
Do you have any recommendations for tools that scan an entire website and report broken inner links?
I run several UGC centered websites and broken inner links, and external, is an issue.
Being that these websites are several hundred thousand pages large, I am not really all that excited about running software on my desktop (xenu link sleuth for example). Any online solutions you could recommend would be great!
-
If it happens to be a wordpress site, there is a plugin called something like "Broken Link Checker." If I recall correctly, that checks internal and outbound links. Otherwise, not too sure.
-
Ideric, did any of these suggestions answer your questions, or have you been able to otherwise find a tool for this? I know others would find the information useful.
At a previous company, we had a custom-written solution to check external links, and made it check response headers until a 200 OK showed, or it got five levels deep. What we'd often find is that we'd have a 301 for an external link, and it'd go from non-www to www. Wouldn't necessarily worry about fixing that, but then later realized that from there, the www link was a 404, OR went to a 200 OK category landing page that said "we've reorganized our site, search here for that individual resource".
-
Well you've found the best solution right here at SEOmoz! Instead of wasting time learning new systems to find out if they'll work or not, just solve your problem. Sign with PRO Elite and you can crawl 100,000 pages.
-
I have used this in the past http://www.auditmypc.com/free-sitemap-generator.asp - (Click on the image in the top right of the instructions) a free tool for site map generation that will show broken internal links in the process. I don't think it has any limits to it, although I have not tried it on a site as large as you are suggesting. Just ensure you are not logged into your site when you run it. Although Google webmaster tools is ok, you can't verify changes made very quickly.
-
I think Xenu is your best option here. The size of the site nearly cuts out the chance a web tool could handle it.
Just recently on a site review I had to run Xenu on a site with 160,000 pages. It only took 4 hours running at 30 threads to complete. Any modern PC should handle it fine.
-
WMT is alright, apart from the fact you can't force Google to crawl all your pages. I would doubt that even a majority of the pages were crawled and indexed by Google (though I don't know what the site is).
Plus, as you say, it only deals with internal links and 404s coming in.
Do you know what the upper limit is on how many crawl errors WMT will display?
-
I might be wrong, but I think Google WMT can accomplish this with ease. I'm looking at 1000 right now. Externally you'll probably have to use xenu =/
-
You might be out of luck on a site that size.
I think WebCEO can do this with their online version but to get 100,000 urls crawled I think it'll cost you a bomb (the sort of money that it'd be cheaper to buy a second PC to run Xenu, lol).
Anyway - http://www.webceo.com/ - I think it may also be possible to install the download version to a server and run it that way.
-
I use Google webmaster tools. Go to diagnostics, then crawl errors.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Link Issues
For several search terms I get site links for the page http://www.waikoloavacationrentals.com/kolea-rentals/kolea-condos/ It makes sense that that page be a site link as it is one of my most used pages, but the problem is google gave it the site link "Kolea 10A". I am having 0 luck making any sense of why that was chosen. It should be something like "Kolea Condos" or something of that nature. Does anyone have any thoughts on where google is coming up with this?
Technical SEO | | RobDalton0 -
Referencing links in Articles and Blogs
Hi I am wondering if the <sup>tag in html is picked up by google as a reference point?</sup> I.e when you put a superscript in word it puts a small number next to your sentence. Then you have a list of reference at the end of the blog/article does google recognise this?
Technical SEO | | Cocoonfxmedia0 -
Tool to Generate All the URLs on a Domain
Hi all, I've been using xml-sitemaps.com for a while to generate a list of all the URLs that exist on a domain. However, this tool only works for websites with under 500 URLs on a domain. The paid tool doesn't offer what we are looking for either. I'm hoping someone can help with a recommendation. We're looking for a tool that can: Crawl, and list, all the indexed URLs on a domain, including .pdf and .doc files (ideally in a .xls or .txt file) Crawl multiple domains with unlimited URLs (we have 5 websites with 500+ URLs on them) Seems pretty simple, but we haven't been able to find something that isn't tailored toward management of a single domain or that can crawl a huge volume of content.
Technical SEO | | timfrick0 -
Are my Canonical Links set up correctly?
I have Enable Canonical Links (recommended) on my web site. However, I also have THIS checked: Enable full URL for Home Page Canonical Link (include /default.asp) Is it hurting me??? Keep getting dinged on our report card. We are using the Volusion shopping cart software/platform.
Technical SEO | | GreenFarmParts0 -
Received Google Webmaster Tools notice of detected unnatural links , but no negative impact on ranking and traffic, What should i do next?
Hello, On May 19 , 2012 Google webmaster sent notification "Google Webmaster Tools notice of detected unnatural links to " both sites haven't lost any ranking or traffic as yet. I am worried, Should i panic,what should i be doing , will it be going down anytime soon? how to naturally build links?
Technical SEO | | conversiontactics0 -
Unnatural links in webmaster tools
Google Webmaster Tools notice of detected unnatural links to my site.I download latest link from google webmaster tool and decided to remove links from last four months.I also got several links from directory sites and i want to remove the links, how to delete links from directory sites ? I f there is no way to delete directory link , please let me know other option to get rid of this issue.
Technical SEO | | Alick3000 -
How to defend against link cloaking
Hi, I own a website where recently a lot of backlinks have been going to my old domain that 301's to my new domain. During the past 2 months I have noticed a massive amount of links pointing to my old domain. When I go to look at the links and go to the page all I see is a search bar which to me this seems like link cloaking. I am not sure what I should do. Obviously I am not doing the link building and someone is targeting anchor specific keywords from multiple domains that all look the same. My question is should I report it myself in google webmaster tools before I get hit with a filter or penalty, or would this force them to penalize me. And if I do get caught up in a penalty I would not know how to fix this since I doubt the webmaster is linking to me out of the kindness of his heart. Any advice? Thanks
Technical SEO | | dreamfire0 -
Do links hold there value after 12 months?
Hello, We need to find out if links that we setup, which are older than 12 months hold any value? Do new links hold more value than old ones and therefore should we let the old links become inactive? If we do let the links become inactive after 12month will that effect the PA/DA of the site?
Technical SEO | | Entrusteddev0