Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New to Affiliates Programs - Any advice?
Hi, I've no background knowledge of affiliate programs and am wondering if this would be a good idea for our company, www.vintageheirloom.com? We sell for the most part expensive, vintage bags and jewellery. Would an affiliates program be appropriate or possibly annoy and hinder customer loyalty rather than promote it? I'm using WP with Woocommerce and have seen a plugin that connects Woocommerce with Ambassador. Does anyone have any previous experience with this company, worked well, any things to watch out for etc. One issue that puzzles me if that I get the idea that you can put a tracking code on say a 3rd party blog banner, person clicks and buy product, that gets tracked. But what if customer abandons sale, comes back later directly to site and purchases? I'm guessing this isn't tracked. Is that right? As an aside I've a blogger friend who I'd like to pay for sending traffic that leads to sales. Is this the best way to track? Would the addition of 'How did you hear about us' question during 'Add to cart' process be a back up method? Any downsides to that, other than customer ignoring it? Thanks a lot for looking & any suggestions you may have! Kevin
Affiliate Marketing | | well-its-1-louder0 -
What is the right way to link to your main site?
Hi, we have a system for tracking the leads that comes from a specific affiliate website. each affiliate has a unique tracking code. not only for affiliates that work with us but also for the SEO team has an affiliate tracking code so our bosses can track the leads and traffic that come from the activity of the SEO team. this means, links towards our website look like: www.mydomain.com/?t_src=campaign&t=AFF&t_cre=links&A=371 www.mydomain.com/?A=371 I have few questions about this: 1. This is the right way to link to my company site? How dose google crawl those link? It's can harm the link value? 2. What is my option to show my boss all the traffic / leads that our seo team brings to the site trough Google Analytics? Hope to get your support. Thanks in advance.
Affiliate Marketing | | JonsonSwartz0 -
Links from Blogs and Forums
Hello Everyone, We are a small, local business in a city in Canada. We just got a new website and I am responsible for basically all of it, including the SEO. I was researching the back-links from our top-ranking competitor (another small, local business in our city). Here is their ranking: PA 40, with 441 Links from 111 RDs DA 30, with 6,864 Links from 116 RDs Here is the second-highest guy: PA 29, with 101 Links from 10 RDs DA 15, with 1,682 Links from 11 RDs As you can see, the top-ranking competitor's PA & DA are just about double his closest competitor. (Don't ask where we currently stand!) But, on closer examination, most of the links from our top-ranking competitor come from a local blog, where our competitor's ad appears in an advertisement on the sidebar. Each time they make a post, which is just about every day, our competitor gets a link. They also have a lot of links from a forum on a rock band's website. The original post was something about finding music, but the responses all have links to unrelated websites. This seems really, really slimy to me. What's going on?
Affiliate Marketing | | StillLearning0 -
Can linking out to a weak site harm my sites SEO?
We have an affiliate that wants us to link to his site to help him get started. His website looks okay and is in the same niche as us. However, he has PR 0 and no Page Trust. We could work in a natural looking link in a news article, but there is no way he could be considered any kind of "authority" link. Can linking to his site harm us?
Affiliate Marketing | | theLotter0 -
Please recommend an affiliate program
I'm a bit surprised there doesn't seem to be a lot of info on their already on SEOMoz, so I'd appreciate any insight into this for a client of mine who is a currently a startup There are lots of options from massive brand names like Google and Amazon right down to countless services I've never heard before. For a smaller start-up with a technical edge. specifically service subscriptions for the webcam / video camera type of market, what would be an effective and cost friendly options to explore? Any shared experience is very appreciated. Here is a list that pretty well captures the top 10, so these are all under consideration: http://www.problogger.net/archives/category/affiliate-programs/ Cheers, Chris
Affiliate Marketing | | cbubinas0 -
Sponsored Links on Amazon, help
So from what I can see CLickrive was discontinued by Amazon, replaced by a9. I have been asked by a client how they can get onto Sponsored Links near the bottom of a search query on an amazon page. Now I can see from the amazon associates site that you can put widgets, display ads, product search links etc but there is no mention of 'Sponsored Links' other than: SPONSORED LINKS Sponsored links are advertisements related to your recent product search query or content on the page. Sponsored links are always clearly labeled. When you click on a sponsored link, we get revenue. Generating additional revenue from sponsored links allows Amazon.com to offer lower prices to you - something we are dedicated to doing in every way we can. There is no mention on how to actually be a sponsored link... I have read on various blogs etc that it is delivered by a third party network... no mention as to whom... thought Google but no mention of this on any information I can find on Google blogs etc. Anyone have an idea? Thank you 🙂
Affiliate Marketing | | ChalkWard0 -
Affilate Links
Hi I’m looking for confirmation of my worries about affiliate links to a website I'm working for and hopefully get some ideas of how to solve the problem of having affiliates without losing the strength of links from affiliate web sites. Poplidays.com has an affiliate program with an independant affiliation platform, but it also has a couple of partners who link directly to Poplidays.com from their sites. Poplidays wants to reward the partner sites in the same way as it does affiliates: with a percentage of sales generated by visitors. However as their SEO I don’t want to lose these links. They are from established, quality sites in the same subject area and, in my opinion, have a lot of SEO value. Affiliate links are 302 redirects from urls on the affiliation platform’s domain; something like http://poplidays.affiliation.com/?clientid=123 to www.poplidays.com Can you confirm that replacing a direct link to www.poplidays.com with a 302 redirect of this type will be a bad for Poplidays rankings? Any proof of this from a reliable source that I can quote? The affiliation platform is insisting that there is no negative effect and that no one has ever made a remark like mine before. Any good ideas of how to resolve the problem of rewarding linkers without spoiling the links for SEO? Bearing in mind that Poplidays want to be able to reward sites for visitors who may buy on a later interaction with the site (up to 30-days) and want an automatic report so that the partner can follow his commissions. I’ve tried implementing something through Google Analytics but it wasn’t perfect. Your in house affiliation system is working with links like http://go.seomoz.org/aff_c?offer_id=1&aff_id=1624&url_id=1&file_id=24 that 302 to http://www.seomoz.org/features?affid=1624&txnid=10274208218121120514 Is this a better / perfect solution ? Compared to a direct link to http://www.seomoz.org/features ? Thanks - Neil
Affiliate Marketing | | CarmenImmobilier0 -
How many affiliate links is considered too many?
Hi, Let's say you have great reviews for 50 products and some of these products do have affiliate links on review pages. And then you have user scores and you come up with a top 20 product list sorted by user scores. Now if you have the list of top 20 products on one page and all these products have an affiliate link (with nofollow relation and a 301 redirect) on the same page with only a couple of images and a summary of the review linked to the review pages, would this still be considered as what Google calls a "bridge page"? Would it be better to still generate the top 20 list but rather link to review pages only? (to avoid too many affiliate links on one single page).
Affiliate Marketing | | Gamer070