Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Person using expired domain and its links to drive traffic
Hi, I know about people using expired domains to drive juice to their primary site but what about people using AN expired domain as their primary site (totally changing that site into a trashy affiliate-marketing vehicle)? The site I'm looking at is thegunzone.com. It has, according to Semrush, almost 38K links. It used to be a legit 17-year-old firearms hobby site, and this is what it originally looked like: http://web.archive.org/web/20120213184627/http://thegunzone.com:80/ Here is its last page before it closed and the domain purchased by the affiliate marketer: http://web.archive.org/web/20170315084035/http://www.thegunzone.com/ It closed around February of 2017, and some affiliate marketer bought it and all its backlinks. However, all those backlinks, which were previously to various articles, are now directed back to those articles (which don't exist anymore) but the homepage, including Wikipedia links. Here's an example: https://en.wikipedia.org/wiki/Polygonal_rifling At the bottom, in the 7th Reference, there's a link to an article called " "Learning About Shooting . . ." but if you click on the original link, it just goes to thegunzone.com homepage. Again, the site's totally different. And there are just thousands of such backlinks to former articles that don't exist anymore but are redirected to this schlocky site's homepage (and it's passing its juice through too). My question is this: this cannot be kosher with Google backlinking policies, right? Is this prevalent on the internet? Why hasn't thegunzone.com been found out and its rankings penalized yet? And how do I report him? I see tons of other sites using this basic strategy too on search results with various hunting keywords. (Disclosure: I do own a hunting/firearms blog, but I don't do any backlinking at all.) Any help would be sincerely appreciated.
Affiliate Marketing | | HandyWoman1 -
What is the danger of integrating different affiliate program widgets far under our core content?
We have pages with lots of unique content that are ranking very well in google.
Affiliate Marketing | | lcourse
I am now considering to add towards the end of the pages widgets from the booking.com affiliate program and also from the affiliate program of two other accommodtion sites. These affilliate widgets would be placed far beyond our core content in the bottom 15% of the page. Each widget will contain a large number of links to the different accomodation options that booking.com or others offer. In our context this information would be indeed very useful for our visitors. Shall I be concerned that our google rankings may suffer due to google perceiving our site as less quality (or google misinterpreting our intent to generate traffic to affiliate programs) ? If google rankings would be suffering, is this something we would normally see in a week (most of our pages are crawled daily by google) and then recover in a week when we would remove the widgets or would this generally be something that would affect the ranking over the course of a month or more? Shall I be concerned abou link dillution since my pages would now have a lot of links pointing to the affiliate sites, which could devalue my internal links? Any thoughts?0 -
What's the best way to go about making Duplicate Content Pages on my domain for affiliates?
Hello, I would like to make a bunch of duplicate pages of my site's Home Page, that way affiliates of mine can have a page of their own with links specialized to themselves littered throughout the page. What's the best practice going about this without jeopardizing domain authority from tons of duplicate content signals firing off?
Affiliate Marketing | | Benavest0 -
Is Affiliate Duplicate Content?
Hi moz fans!
Affiliate Marketing | | KasperK
I'm working on some client sites.
They have a lot of products, which they are selling through their webshop and two affiliate sites.
But I can't stop wondering:
The affilate sites are using the same product descriptions and images - Will Google see that as duplicate content?
And if so.. How can I prevent that? The two affiliate sites are well etablished and huge sites, so I guess they know what they are doing.. Is there some way I can check it? When I do a google search, whit the "My product description text", I'm getting some different results: Sometimes all three sites appear (main site, and affilate) Sometimes it's only the affilates Sometimes it's only the main site. So to make it short:
If i'm using affliate, will Google see it as duplicate content?
How do I check it, and how do I prevent it? Thank you!0 -
How to track Affiliate Clicks to Google Analytics
Hi, We would like to track the affiliate link clicks. Is there any way to track it from Google Analytic? Rajiv
Affiliate Marketing | | gamesecure0 -
How to setup affiliate marketing for my website?
I work for an ecommerce company and would like to setup some sort of affiliate marketing similar to Amazon. 1. If someone was referred to our site and they made a purchase we would pay out. 2. If we referred a user to someone else's page and they made a purchase we would get paid. I looked into http://www.commissionfactory.com/ but wasn't sure if that would meet my needs. Does anyone have any recommendations?
Affiliate Marketing | | EcommerceSite0 -
Affiliate Programs
Hi, I am wondering what if any affiliate management applications are being used out there. Anyone using Shareasale... I am looking into them.
Affiliate Marketing | | unikey0 -
Does Google look at Domain Registrar owner information when counting links into a site?
Question: If the domain registrar owner info is the same on two websites, will Google/other search engines discredit any linking between the two sites when calculating page rank? We have two company sites. One is our main ecommerce domain which we have been linking into from information domains in the past. We recently cut ties with the company that hosted them for us and wish to host them ourselves to preserve the inbound traffic. We are worried, however, that the traffic will be discredited now that we own both domains. Thanks for looking!
Affiliate Marketing | | tennisexpress0