Any tools for scraping blogroll URLs from sites?
-
This question is entirely in the whitehat realm...
Let's say you've encountered a great blog - with a strong blogroll of 40 sites.
The 40-site blogroll is interesting to you for any number of reasons, from link building targets to simply subscribing in your feedreader. Right now, it's tedious to extract the URLs from the site. There are some "save all links" tools, but they are also messy.
Are there any good tools that will
a) allow you to grab the blogroll (only) of any site into a list of URLs (yeah, ok, it might not be perfect since some sites call it "sites I like" etc.)
b) same, but export as OPML so you can subscribe.
Thanks!
Scott
-
Not at all. I guess my feeling here is that there is a sort of untapped social graph defined by blogrolls. If it were simple to harvest them upon visiting a blog (e.g. this blogger recommends...) one could do a stumble-on-steroids approach to a niche.
-
I thought you might be able to use the outbound link scraper to grab the outbound link onto the page. Pop in your URLS of the pages you want to scrape and it will spit out our a list of those domaind and urls. You can take those urls and put them into the contact finder and it will return the contact details for those sites. Combine the two spreadsheets for an epiuc list of blogs to contact for your outreach.
This is obviously for link building rather than subscribing - sorry if I have misunderstood what you were trying to do
-
Hi Keri,
That is a very cool tool, but is overkill for this. It takes far too many steps to accomplish only part of the desired goal of grabbing all blogroll URLs (within the blogroll DIV tag) and exporting the list to a valid OMPL file or URL list.
thanks!
-
nothing I saw there would do this. It looks like it could manage to list all external links, and I suppose you could manually pick the blogroll out of it.
-
Hi there,
Well, Keris response reminded me of this question and the fact that I found a tool for scraping these kind of lists:
Here it is (with some other cool tools) , have fun:
-
Hi Scott,
I'm going through older questions. Did you ever find a tool to do what you wanted to do here?
-
One thing to look at is Outwit Hub for Firefox. It might be able to help with that. It can scrape data from a page and do a lot with it. http://www.outwit.com/products/hub/. Don't know that it meets all of your needs, but I also haven't seen a response with anything better at the moment.
-
Hey Scott,
What a great question and <sigh>I don't have the answer. I am going to back to find out what people come up with here. Surely there is someone that lurks these parts that can throw something together?</sigh>
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Difference between urls and referring urls?
Sorry, nit new to this side of SEO We recently discovered we have over 200 critical crawler issues on our site (mainly 4xx) We exported the CSV and it shows both a URL link and a referring URL. Both lead to a 'page not found' so I have two questions? What is the difference between a URL and a referring URL? What is the best practice/how do we fix this issue? Is it one for our web developer? Appreciate the help.
Moz Pro | | ayrutd1 -
Organic Traffic Drops to Zero After Site Migration
I know this question is terribly general . . . but I don't even know where to start in terms of sharing info, which I would be glad to post on request. My site was recently re-designed and migrated to a new server. I certainly expected SOME drop in traffic as google gets the snew site all figured out, but as you can see this drop has been dramatic. It's a simple wordpress site, I have yoast ptimized on it, I don't have many obvious errors that have not already been fixed that I can see.
Moz Pro | | damon1212
I just . ..what the heck is going on and how can I fix it? Is it normal and takes a month or two to sort out? Help! Screen%20Shot%2012-04-15%20at%2011.32%20AM_zpsczdmzcjt.png0 -
Crawl Errors from URL Parameter
Hello, I am having this issue within SEOmoz's Crawl Diagnosis report. There are a lot of crawl errors happening with pages associated with /login. I will see site.com/login?r=http://.... and have several duplicate content issues associated with those urls. Seeing this, I checked WMT to see if the Google crawler was showing this error as well. It wasn't. So what I ended doing was going to the robots.txt and disallowing rogerbot. It looks like this: User-agent: rogerbot Disallow:/login However, SEOmoz has crawled again and it still picking up on those URLs. Any ideas on how to fix? Thanks!
Moz Pro | | WrightIMC0 -
Very confused on site.com/ or not using a /
I'm wanting to put the rel="canonical" tag on my homepage but I'm not sure which to use? How would you know what to use and always links to, http://www.site.com or http://www.site.com**/** Personally I never knew there was a difference until I used the seomoz tool and I wasn't using the tag.
Moz Pro | | GYMSN0 -
What analysis exists for Out Bound Links (OBL) from your site
Hi Maybe I am missing this but I can''t seem to see it. I am doing some analysis on a client's site and want to get a csv list of links from the client's site to external sites. So what I am looking for is a list of Out Bound Links (OBL) from the client's site. I want to run these past a black list / bad link neighborhood checking script I have. This would actually be a nice feature in SEO Moz Pro, unless it actually already does and I am just missing it or not setting filters correctly. Thanks Trevor
Moz Pro | | tstolber10 -
Best Keyword Ranking Report Tool
What's the best keyword ranking report tool that anyone is using currently? I have used CuteRank (booo), Rank Tracker, SEOBook.com Rank Tracker, etc. Cost isn't an issue - I'm looking for a free or a paid tool. I need to be able to export resutls in PDF and Excel format. Thanks for the feedback!
Moz Pro | | ezclickmedia0 -
Which tools are better? SEOMoz Tools or Bruce Clay's Tools.
I've ALWAYS wanted to hear some discussion on this, please give me your honest opinion so I can make the correct decision.
Moz Pro | | fergseo2 -
Tool for scanning the content of the canonical tag
Hey All, question for you. What is your favorite tool/method for scanning a website for specific tags? Specifically (as my situation dictates now) for canonical tags? I am looking for a tool that is flexible, hopefully free, and highly customizable (for instance, you can specify the tag to look for). I like the concept of using google docs with the import xml feature but as you can only use 50 of those commands at a time it is very limiting (http://www.distilled.co.uk/blog/seo/how-to-build-agile-seo-tools-using-google-docs/). I do have a campaign set up using the tools which is great! but I need something that returns a response faster and can get data from more than 10,000 links. Our cms unfortunately puts out some odd canonical tags depending on how a page is rendered and I am trying to catch them quickly before it gets indexed and causes problems. Eventually I would also like to be able to scan for other specific tags, hence the customizable concern. If we have to write a vb script to get it into excel I suppose we can do that. Cheers, Josh
Moz Pro | | prima-2535090