Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Affiliate Tool For Competitors Analysis
Hello Guys, I'm Looking for an Affiliate Tool where i can see my competitors affiliate publisher. is there any kind of tool which can help me out to find the best publishers of my competitors? Please Suggest. Thanks Damian Martin
Affiliate Marketing | | CommercePundit0 -
Affiliate marketing: yes or no?
Hello, A colleague recently asked me about affiliate marketing for our clients, and my answer for a few years now has been emphatically "no", because I had not seen it done in positive way. How does everyone weigh in affiliate marketing, and what companies and/or software would you suggest? For example, is http://www.admedia.com/ a reputable source? I see lots of bad reviews on these guys. Any thoughts, suggestions, etc would be extremely valuable. Thanks in advance!
Affiliate Marketing | | lfrazer0 -
Advise about Social Sharing between two loosely affiliated website - (retweets/fb cross sharing)
So basically, I have two websites. One being my Primary website and another which is very Niche/Seasonal. Now the Niche website received a penalty, and was later re-included however google traffic has been extremely limited since being reindexed. (bing never had a problem, still top ranked) I don't directly link back and forth between the two websites because its not worth hurting my primary websites ranking. I'm sure google knows I own both anyway, as they are on the same webmaster tools but the Niche site was kicked for spammy link building & its been an endless process removing them all. Anyway, with that being my concern. Would it be acceptable to cross share facebook posts, retweet and do things like that on occasion? My primary reason for asking is because the Niche website has a very large social following, so I would like to share post/tweets from my Primary site when its relevant and vice versa. Appreciate any suggestions/comments. Thanks!
Affiliate Marketing | | Southbay_Carnivorous_Plants0 -
Affiliate URLs Indexed in Google
We have an active in-house affiliate program which has a create-a-link function where affiliate, as in most affiliate programs, can build links back to our content to produce sales. The problem that I am seeing is that some of the affiliate URL versions are being indexed in Google rather than the original page. For instance.. http://www.ourdomain.com/page_start.aspx?affnum=F025212&Start_Page=RaceCar&referrer=createlink is outranking... http://www.ourdomain.com/racecar This presents 2 issues for us. First, this presents duplicate content issues obviously. Second, we pay our affiliates a portion of the sales so, in order to maximize profitability, we'd like the indexed pages to be the original version. What's the best way to handle this? I want to make sure that affiliates continue to get credit for links from their sites but search engine links should go to the original versions.
Affiliate Marketing | | ATIseo0 -
In what ways would an affiliate site be okay with Google?
I know Google has slammed affiliate sites hard. But affiliate marketing can be a real business and had an idea for one, but I would like to know if ALL affiliate sites are looked at as evil in Google's eyes, or if it's just those with thin and duplicate content in the product descriptions. I know they want brands to come up ahead of affiliates of that brand, and that makes sense. If I sold Acme Widgets products, Acme Widgets itself should come up ahead of my site. But what if the site sold widgets in general, with the term widgets not a branded keyword? If all product descriptions are unique and well-written content, on a site that is high quality, would it still be downgraded by Google just because there are affilate links in it? I guess overall the question could be boiled down to, are affiliate sites hit because they have horrible content or because they have links to affiliate programs in them?
Affiliate Marketing | | bizzer0 -
Affiliates content sharing
Quick question: Is it ok to provide our product descriptions to affiliates? Ho does this effect our SEO as far as duplicate content issue? Best Regards
Affiliate Marketing | | Jvalops0 -
How much SEO benefit am I getting from 'affiliate' links?
Hi all, We run lots of different campaigns with affiliates and as such they have links on their websites pointing back to ours. I was keen to know how much 'link-juice' these kinds of links are passing back to our websites or how much SEO benefit we are getting from them. The links have "?=affiliate" in them. Any thoughts would be greatly appreciated!! Many thanks! Rob
Affiliate Marketing | | RobertHill0 -
In search of the perfect SEO affiliate ID'd URL
Hi, I'm building our own affilaite system. I want to make sure I get as much SEO benifit from affiliate links as possible so I am in search of the perfect solution. My thoughts after doing a bit of research are to do the following - using the # as my identifier: 1/. Use an ID like www.mysite.com/#A123 2/. Redirect via 301 these urls to www.mysite.com Any thoughts on this, or other things I should be thinking of with regard so building an affilaite system that has strong SEO benifits? Thanks
Affiliate Marketing | | James770