Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Affiliate links and parameters creating duplicate page titles
Hi, we have an affiliate system and use rel=**canonical **to canonicalise them to the right link. Still, we see those links creating duplicate page titles, in Moz crawl and in Google search console. How are we supposed to treat those links so they do not create duplicate content? Thanks for your help!
Affiliate Marketing | | guidetoiceland0 -
Is Affiliate Duplicate Content?
Hi moz fans!
Affiliate Marketing | | KasperK
I'm working on some client sites.
They have a lot of products, which they are selling through their webshop and two affiliate sites.
But I can't stop wondering:
The affilate sites are using the same product descriptions and images - Will Google see that as duplicate content?
And if so.. How can I prevent that? The two affiliate sites are well etablished and huge sites, so I guess they know what they are doing.. Is there some way I can check it? When I do a google search, whit the "My product description text", I'm getting some different results: Sometimes all three sites appear (main site, and affilate) Sometimes it's only the affilates Sometimes it's only the main site. So to make it short:
If i'm using affliate, will Google see it as duplicate content?
How do I check it, and how do I prevent it? Thank you!0 -
Affiliate Marketing in Australia
Hello Guys My website is BannerBuzz.com.au I'm Looking for Best Affiliate Marketing Websites In Australia and I've Researched and found the following Websites for the same. http://www.clixgalore.com.au/
Affiliate Marketing | | CommercePundit
http://www.jackmedia.com/
http://www.commissionfactory.com.au/ So I Need your valuable opinions on which website I can get maximum exposure in Affiliate Marketing. Please Suggest. Thanks in Advance. Thanks Prabhakar0 -
What is the best way to boost no. of quality sites linking in?
Analysing my site ranking compared to competitors, I see that our weakness is getting others to share our content and link into it via their own blogs and social media - although I would say our social media activity is pretty good. We have also worked on getting listed free on relevant sites related to our product area (learn spanish in spain) in order to increase sites linking but this is still minute compared to our 2 main competitors. (As a note: paid listings are not totally justifiable for us at this point in time but perhaps in the future we will expand on this). Am I missing something here?
Affiliate Marketing | | sparkit0 -
Does having affiliates hurt seo?
one of my clients has a TON of affiliates out there building links back to his site all with the ?ref=AFFILIATENAME tracking string does google understand this and not count these links or is there a way to nofollow all links with this tracking string? just wondering how big companies rank well and still have thousands of affiliates building links on all kinds of properties using all kinds of methods thanks
Affiliate Marketing | | Ezpro90 -
Recommended affiliate programmes (particularly UK-based)?
I'm starting a photography/travel/music blog and thought it might be a good idea to stick the occasional affiliate link in when it's appropriate. Does anyone have any recommendations? The related items would mainly be to DVDs, books and music, and sometimes something more travel specific like flights. Are the programmes from the likes of Amazon and ebay good, or would it be better to sign up to a system that covers many companies? It'll be my own CMS if that makes any difference. Thanks. 🙂
Affiliate Marketing | | Alex-Harford0 -
Affilate Links
Hi I’m looking for confirmation of my worries about affiliate links to a website I'm working for and hopefully get some ideas of how to solve the problem of having affiliates without losing the strength of links from affiliate web sites. Poplidays.com has an affiliate program with an independant affiliation platform, but it also has a couple of partners who link directly to Poplidays.com from their sites. Poplidays wants to reward the partner sites in the same way as it does affiliates: with a percentage of sales generated by visitors. However as their SEO I don’t want to lose these links. They are from established, quality sites in the same subject area and, in my opinion, have a lot of SEO value. Affiliate links are 302 redirects from urls on the affiliation platform’s domain; something like http://poplidays.affiliation.com/?clientid=123 to www.poplidays.com Can you confirm that replacing a direct link to www.poplidays.com with a 302 redirect of this type will be a bad for Poplidays rankings? Any proof of this from a reliable source that I can quote? The affiliation platform is insisting that there is no negative effect and that no one has ever made a remark like mine before. Any good ideas of how to resolve the problem of rewarding linkers without spoiling the links for SEO? Bearing in mind that Poplidays want to be able to reward sites for visitors who may buy on a later interaction with the site (up to 30-days) and want an automatic report so that the partner can follow his commissions. I’ve tried implementing something through Google Analytics but it wasn’t perfect. Your in house affiliation system is working with links like http://go.seomoz.org/aff_c?offer_id=1&aff_id=1624&url_id=1&file_id=24 that 302 to http://www.seomoz.org/features?affid=1624&txnid=10274208218121120514 Is this a better / perfect solution ? Compared to a direct link to http://www.seomoz.org/features ? Thanks - Neil
Affiliate Marketing | | CarmenImmobilier0 -
Affilate links - Google Panda 2.1
With the recent update from Google Panda would links from affiliate Coupon Sites now have an negative impact in Organic Rankings or site value? A reason for this question is because many Coupon Sites usually have poor content and a poor quality score. Thanks
Affiliate Marketing | | Tonyd230