Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My affiliate links are being crawled by Google. Is this a bad thing? Also, should I add rel: nofollow to affiliate links?
Hi All, I have an issue on Google Search Console. It seems all my affiliate links are being crawled. Is this a bad thing? The reason I found out is that all but one link was flagged with a warming. It goes on to say that it was blocked by robots.txt. The strange thing is that I've not set any links to be blocked in robots.txt. Also, should affiliate links be nofollow? I'm asking because I was recently sent this article by Yoast on how to properly cloak my affiliate links without using a plugin. https://yoast.com/cloak-affiliate-links/ In the method, they apply the nofollow and noindex rules to the links. I'm just a little confused on how to go about affiliate links the SEO friendly way.
Affiliate Marketing | | nomad_blogger0 -
Blocking external links in Robots.txt - need advice on Best Practice
I look after an affiliate site that is doing quite well in the search rankings. We've been doing a review of our practices and one thing that has cropped up is our robots.txt. In it, we disallow Google from crawling external links. This used to be best practice in the affiliate industry a couple of years ago, but I wanted to know if this is still the case, and what the implications are if we were to: a/ leave it as is? b/ allow crawling? Thanks in advance.
Affiliate Marketing | | Ben_Malkin_Develo0 -
Is Affiliate Duplicate Content?
Hi moz fans!
Affiliate Marketing | | KasperK
I'm working on some client sites.
They have a lot of products, which they are selling through their webshop and two affiliate sites.
But I can't stop wondering:
The affilate sites are using the same product descriptions and images - Will Google see that as duplicate content?
And if so.. How can I prevent that? The two affiliate sites are well etablished and huge sites, so I guess they know what they are doing.. Is there some way I can check it? When I do a google search, whit the "My product description text", I'm getting some different results: Sometimes all three sites appear (main site, and affilate) Sometimes it's only the affilates Sometimes it's only the main site. So to make it short:
If i'm using affliate, will Google see it as duplicate content?
How do I check it, and how do I prevent it? Thank you!0 -
How to track Affiliate Clicks to Google Analytics
Hi, We would like to track the affiliate link clicks. Is there any way to track it from Google Analytic? Rajiv
Affiliate Marketing | | gamesecure0 -
Affiliate marketing: yes or no?
Hello, A colleague recently asked me about affiliate marketing for our clients, and my answer for a few years now has been emphatically "no", because I had not seen it done in positive way. How does everyone weigh in affiliate marketing, and what companies and/or software would you suggest? For example, is http://www.admedia.com/ a reputable source? I see lots of bad reviews on these guys. Any thoughts, suggestions, etc would be extremely valuable. Thanks in advance!
Affiliate Marketing | | lfrazer0 -
How do you find Affiliate Links on your site that have not been nofollowed?
We've just signed up to an affiliate scheme because we were sending links to them because we thought their product was valuable to our users. So we now have to go through and nofollow all of these links over 100's of pages. Is there any way that do a crawl of the site to identify all links to a particular site and tell me what page they are on and whether they are nofollow/follow?
Affiliate Marketing | | Zippy-Bungle0 -
Affiliate URLs Indexed in Google
We have an active in-house affiliate program which has a create-a-link function where affiliate, as in most affiliate programs, can build links back to our content to produce sales. The problem that I am seeing is that some of the affiliate URL versions are being indexed in Google rather than the original page. For instance.. http://www.ourdomain.com/page_start.aspx?affnum=F025212&Start_Page=RaceCar&referrer=createlink is outranking... http://www.ourdomain.com/racecar This presents 2 issues for us. First, this presents duplicate content issues obviously. Second, we pay our affiliates a portion of the sales so, in order to maximize profitability, we'd like the indexed pages to be the original version. What's the best way to handle this? I want to make sure that affiliates continue to get credit for links from their sites but search engine links should go to the original versions.
Affiliate Marketing | | ATIseo0 -
Can anyone help me find the broadband ISP affiliate program I need?
I have a couple sites that deal with broadband internet access, and I'm looking to monetize beyond adsense. I'd like to sign up for some affiliate programs from internet service providers like Charter, CenturyLink, Comcast, etc. I found a couple affiliate programs but I am having no luck at all figuring out the programs that so many other websites are involved with. In about every case, the user is able to enter in their address and then a search is performed, returning programs available at their location. The programs I've found just do the traditional ad click-throughs for a possible commission. Can anyone enlighten me what kind of programs they are using?
Affiliate Marketing | | bizzer0