Blocking external links in Robots.txt - need advice on Best Practice
-
I look after an affiliate site that is doing quite well in the search rankings. We've been doing a review of our practices and one thing that has cropped up is our robots.txt.
In it, we disallow Google from crawling external links. This used to be best practice in the affiliate industry a couple of years ago, but I wanted to know if this is still the case, and what the implications are if we were to:
a/ leave it as is?
b/ allow crawling?
Thanks in advance.
-
I'm not sure how you're blocking Google from crawling external links in the robots.txt file--typically you only block them from crawling internal pages on your site.
If you're using a script, though, to track the clicks on external links and that script is running your site (and you're blocking that script in robot.txt), then that still should be fine. You may want to add a "nofollow" tag on those links, though, so you don't end up passing link "credit" or "link juice" to those affiliates (unless you want to do that?).
As far as external links go, though, it's typically okay (and expected) that you link out to other sites.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Spent months creating links but Webmasters not picking it up
Hey Guys, I've been working on Blast My Deals.com an affiliate site which i have an issue hoping you guys can help with. For the past year and a half we created loads of links from multiple domains at-least over 700+ which Webmasters did pick up since then we've created even more links however when we go in to our Search Console "Links to your site" it shows 214 domains. It seems like our old links are being removed and our new links aren't being added. All the links we created have an average of DA40+ For the past 2 weeks ive noticed a significant drop in organic traffic and loosing our positions on our main keywords. Our DA has also dropped from 33 to 29 in a matter of 2 weeks. I checked everything including robots.txt, sitemap.xml but Im unable to find the reason for it.
Affiliate Marketing | | kazirahman0 -
Which is the Best Affiliate Marketing Platform in Australia?
We have 1 eCommerce website (www.bannerbuzz.com.au) for printing products like banners, signs, banner stands, teardrop flags and more... We want to start affiliate marketing for our products with CPA model. So, Do anyone have a good experience with Australian affiliate marketing platform. We are having shareasale account for our USA based website, But Shareasale is not that much aggressive in Australia. So we want to finalize 1 platform for affiliate marketing which can have good strength of advertisers and reputation. Thanks,
Affiliate Marketing | | CommercePundit0 -
Find all affiliate links in a custom CMS
I have a custom CMS. I offer a few affiliate products through my site. Now I need to update all my aff. links and I'm not sure how I can find all the pages (posts) on my site that have affiliate links. What's the best way to do that?
Affiliate Marketing | | VinceWicks0 -
What is the negative effect of getting affiliate links to your domain?
Are the negative effects of affiliate links to your domain bigger than the positive effects? Is it better to use a subdomain for your affiliate campaigns? (Because of linkprofile no follow links or time on page effects?) Thanks for responce!
Affiliate Marketing | | ltom0 -
Can anyone help me find the broadband ISP affiliate program I need?
I have a couple sites that deal with broadband internet access, and I'm looking to monetize beyond adsense. I'd like to sign up for some affiliate programs from internet service providers like Charter, CenturyLink, Comcast, etc. I found a couple affiliate programs but I am having no luck at all figuring out the programs that so many other websites are involved with. In about every case, the user is able to enter in their address and then a search is performed, returning programs available at their location. The programs I've found just do the traditional ad click-throughs for a possible commission. Can anyone enlighten me what kind of programs they are using?
Affiliate Marketing | | bizzer0 -
Link Building Intricacies
I'm about to embark on link building campaign. I have several key phrases that I want to develop such as "health & fitness products" and "vitamins & supplements". My site is specifically targeted at Australia men. Should I also aim to include the words "Australia" and "men" in my link anchor text? Or should leave the qualifiers to the title tag for my destination url? I'm also wondering, by having "health & fitness products" as my link anchor text and the same text in my title tag for my destination url, will that build value for the various keyword variations in my keyword phrase such as "health & fitness" and "fitness products"? Thanks!
Affiliate Marketing | | B-man0 -
Link juice from my affiliates to my site
I'm setting up an affiliate program through 1shoppingcart. Even though I run 3 different websites, the way their program works is that I have to have 1 domain name that my affiliates get as their affiliate link, with their unique id attached at the end. I can give them multiple links that will redirect to various places on my 3 websites, but those original links that they put on their site will all be to just 1 domain. I'm wondering if I still get link juice to my sites even though the links are obviously being redirected? Does it depend on the redirection method? I was just thinking that getting hundreds of affiliates who run related sites could be a very good seo boost if they're all linking to my site, but I feel like it's going to be wasted because of this redirection thing. Hope this made sense. Thanks very much,
Affiliate Marketing | | philraymond
Phil0