Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is Affiliate Duplicate Content?
Hi moz fans!
Affiliate Marketing | | KasperK
I'm working on some client sites.
They have a lot of products, which they are selling through their webshop and two affiliate sites.
But I can't stop wondering:
The affilate sites are using the same product descriptions and images - Will Google see that as duplicate content?
And if so.. How can I prevent that? The two affiliate sites are well etablished and huge sites, so I guess they know what they are doing.. Is there some way I can check it? When I do a google search, whit the "My product description text", I'm getting some different results: Sometimes all three sites appear (main site, and affilate) Sometimes it's only the affilates Sometimes it's only the main site. So to make it short:
If i'm using affliate, will Google see it as duplicate content?
How do I check it, and how do I prevent it? Thank you!0 -
Setting up external link goals in GA
I have recently switched all my travel affiliate links to TripAdvisor using their javascript code that converts the natural links (direct links to TA URLs) to affiliate links. I have set up goal tracking in GA to try to put a monetary value on clicks through to these links (based on the known conversion rate to commissionable bookings) but GA is over-reporting the goals by a very large margin. I'm wondering what I've done wrong. The first screen shot below is of the goal settings I used in GA. The second screenshot shows that GA is reporting 7125 goal completions since I set up the goal on 7 August. The third screenshot shows the actual number of GA events recorded on outbound clicks to TA URLs in the same period, which is just 97. So the goal completions being recorded far exceed the actual goal completions, but I'm not sure why. I know I've done something wrong but as this is the first time I've used goals, I'd appreciate some help to see what I've done wrong. Thank you. 3f7dc3de6f9f1373d47994144a5cb016 56aa39130a7d54b07d8f2ce9cf1d7fb0 35ee827275928ca8c4d4c941f0672ef8
Affiliate Marketing | | Gavin.Atkinson0 -
Can linking out to a weak site harm my sites SEO?
We have an affiliate that wants us to link to his site to help him get started. His website looks okay and is in the same niche as us. However, he has PR 0 and no Page Trust. We could work in a natural looking link in a news article, but there is no way he could be considered any kind of "authority" link. Can linking to his site harm us?
Affiliate Marketing | | theLotter0 -
Duplicate tracking of Affiliate and Organic orders
I started diving into the over lap between Organic and affiliate orders and can see that on average about 10% of my organic orders get tracked as both affiliate and organic (meaning i am paying affiliate commissions on them). I want to be able to better account for the different types of order and where they are coming from. My question is simply if anyone has any ideas as to how better to figure out if those orders should be canceled on the affiliate end or if there is a better way in analytics to see where the order is actually getting attributed to through some type of goal or advanced set up or something (the download of orders from organic that I compared to the affiliate ones was taken from Ecommerce > Transactions > Medium (Organic) then I compared that to the order numbers from the affiliate company I am using). Hope this makes sense and thanks for any information that can help with this!
Affiliate Marketing | | Gordian0 -
Looking for a nice way to add an amazon affiliate store to my site
We are often asked what our favorite toys, books, resources, etc, are and I want to make a page(s) with our top resources that is a nice clean amazon affiliate store. Are there are good resources anyone knows of to do this? (I'm using a wordpress blog by the way....perhaps there is a plug in?) Thanks.
Affiliate Marketing | | NoahsDad0 -
Will giving a blogger an affiliate url to my site reduce her links' seo value to me?
If a blogger were to write a post about one of my products and link to my product page, would giving her an affiliate url to link to rather than the normal link spoil the seo value of the link? So instead of mysite.com/product-one it would be mysite.com/affiliate/blogger/product-one Google would think... this is a transaction not a vote of confidence?
Affiliate Marketing | | Brocberry0 -
Affiliate System for my Website
So, I like to implement one affiliate system on my website and to generate URL for every online marketing campaign. because i like to know when someone has ordered it from Organic, and when from any Google Adwords Campaigns etc.. What do you suggest me to implement?
Affiliate Marketing | | leadsprofi0 -
Important question on affiliate links
The website I am working on is http://www.treadmillreviews.com. If you look at any of our affiliate links, they all have an href value of # or something that keeps the user on the page. However, there is an onclick="window.open('affiliate link URL')" code in them that has the affiliate link open in a new window. This website was set up to "hide" the affiliate links in essence this way. I just tried to make a PPC campaign in Google. Got shut down in about 20 minutes for being a "bridge page." Now, all of the content is original and not copied from anywhere. Supposedly Google could not see these links. Yet we got labeled how we did, and now I"m thinking it is causing rankings to hurt a bit. My question is, should I change how all of the links are set up? Should I make the href value be the affiliate link? It is a redirect like CJ.com uses. Should I nofollow those affiliate links? What is the best way to handle this problem? All help is appreciated.
Affiliate Marketing | | DanDeceuster0