Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • Case Studies
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      What is your Brand Authority?
      Moz

      What is your Brand Authority?

      Check yours now
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • SEO Q&A

        Insights & discussions from an SEO community of 500,000+.

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • Case Studies

        Explore how Moz drives ROI with a proven track record of success.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. Digital Marketing
    3. Affiliate Marketing
    4. Looking for a way to crawl and test validity of affiliate links at scale. Ideas?

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Looking for a way to crawl and test validity of affiliate links at scale. Ideas?

    Affiliate Marketing
    2
    4
    1978
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • BradsDeals
      BradsDeals last edited by

      Hey all,

      I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?

      Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.

      Bi75jku

      1 Reply Last reply Reply Quote 0
      • BradsDeals
        BradsDeals @Everett last edited by

        So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.

        1 Reply Last reply Reply Quote 1
        • BradsDeals
          BradsDeals @Everett last edited by

          Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.

          1 Reply Last reply Reply Quote 1
          • Everett
            Everett last edited by

            Hello Rebecca,

            Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.

            If that doesn't work, have you tried Deep Crawl?

            BradsDeals 2 Replies Last reply Reply Quote 1
            • 1 / 1
            • First post
              Last post

            Got a burning SEO question?

            Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


            Start my free trial


            Browse Questions

            Explore more categories

            • Moz Tools

              Chat with the community about the Moz tools.

            • SEO Tactics

              Discuss the SEO process with fellow marketers

            • Community

              Discuss industry events, jobs, and news!

            • Digital Marketing

              Chat about tactics outside of SEO

            • Research & Trends

              Dive into research and trends in the search industry.

            • Support

              Connect on product support and feature requests.

            • See all categories

            Related Questions

            • lcourse

              What is the danger of integrating different affiliate program widgets far under our core content?

              We have pages with lots of unique content that are ranking very well in google.
              I am now considering to add towards the end of the pages widgets from the booking.com affiliate program and also from the affiliate program of two other accommodtion sites. These affilliate widgets would be placed far beyond our core content in the bottom 15% of the page. Each widget will contain a large number of links to the different accomodation options that booking.com or others offer. In our context this information would be indeed very useful for our visitors. Shall I be concerned that our google rankings may suffer due to google perceiving our site as less quality (or google misinterpreting our intent to generate traffic to affiliate programs) ? If google rankings would be suffering, is this something we would normally see in a week (most of our pages are crawled daily by google)  and then recover in a week when we would remove the widgets or would this generally be something that would affect the ranking over the course of a month or more? Shall I be concerned abou link dillution since my pages would now have a lot of links pointing to the affiliate sites, which could devalue my internal links? Any thoughts?

              Affiliate Marketing | | lcourse
              0
            • Benavest

              What's the best way to go about making Duplicate Content Pages on my domain for affiliates?

              Hello, I would like to make a bunch of duplicate pages of my site's Home Page, that way affiliates of mine can have a page of their own with links specialized to themselves littered throughout the page. What's the best practice going about this without jeopardizing domain authority from tons of duplicate content signals firing off?

              Affiliate Marketing | | Benavest
              0
            • edward-may

              Affiliate Program: Does anyone know how to set this up?

              Hi Guys Does anyone know how I can make a affiliate programme on my website eg - I have a sales person in two of our local regions and I want to find a way I can track any sales made through them via my website so I can pay them their commission. Any advice around this, the website is wordpress! Cheers Justin

              Affiliate Marketing | | edward-may
              0
            • lfrazer

              Affiliate marketing: yes or no?

              Hello, A colleague recently asked me about affiliate marketing for our clients, and my answer for a few years now has been emphatically "no", because I had not seen it done in positive way. How does everyone weigh in affiliate marketing, and what companies and/or software would you suggest? For example, is http://www.admedia.com/ a reputable source? I see lots of bad reviews on these guys. Any thoughts, suggestions, etc would be extremely valuable. Thanks in advance!

              Affiliate Marketing | | lfrazer
              0
            • unikey

              Affiliate Programs

              Hi, I am wondering what if any affiliate management applications are being used out there. Anyone using Shareasale... I am looking into them.

              Affiliate Marketing | | unikey
              0
            • SeeSharp1

              Amazon Links destroyed my rankings, I removed them, will my site recover?

              Hello, Last week I got the glorious idea to put amazon affiliate links on my website. I didn't put a rel=nofollow on them or deindex the pages they were on. Precisely I put some reviews and 10 different amazon links on one page and I put that into my navigation so all inner pages were linking to that page. When Google crawled my page again, I had a 60% traffic drop. 3 days later when I found out what caused this (amazon links) I completely removed the site containing those links and removed all links to that page from my site. Now my rankings are still down, will my site recover from this or is it lost? And how long would you think it'll take?

              Affiliate Marketing | | SeeSharp1
              0
            • tennisexpress

              Does Google look at Domain Registrar owner information when counting links into a site?

              Question: If the domain registrar owner info is the same on two websites, will Google/other search engines discredit any linking between the two sites when calculating page rank? We have two company sites. One is our main ecommerce domain which we have been linking into from information domains in the past. We recently cut ties with the company that hosted them for us and wish to host them ourselves to preserve the inbound traffic. We are worried, however, that the traffic will be discredited now that we own both domains. Thanks for looking!

              Affiliate Marketing | | tennisexpress
              0
            • Gamer07

              How many affiliate links is considered too many?

              Hi, Let's say you have great reviews for 50 products and some of these products do have affiliate links on review pages. And then you have user scores and you come up with a top 20 product list sorted by user scores. Now if you have the list of  top 20 products on one page and all these products have an affiliate link (with nofollow relation and a 301 redirect) on the same page with only a couple of images and a summary of the review linked to the review pages, would this still be considered as what Google calls a "bridge page"? Would it be better to still generate the top 20 list but rather link to review pages only? (to avoid too many affiliate links on one single page).

              Affiliate Marketing | | Gamer07
              0

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy

            Looks like your connection to Moz was lost, please wait while we try to reconnect.