Blocking external links in Robots.txt - need advice on Best Practice
-
I look after an affiliate site that is doing quite well in the search rankings. We've been doing a review of our practices and one thing that has cropped up is our robots.txt.
In it, we disallow Google from crawling external links. This used to be best practice in the affiliate industry a couple of years ago, but I wanted to know if this is still the case, and what the implications are if we were to:
a/ leave it as is?
b/ allow crawling?
Thanks in advance.
-
I'm not sure how you're blocking Google from crawling external links in the robots.txt file--typically you only block them from crawling internal pages on your site.
If you're using a script, though, to track the clicks on external links and that script is running your site (and you're blocking that script in robot.txt), then that still should be fine. You may want to add a "nofollow" tag on those links, though, so you don't end up passing link "credit" or "link juice" to those affiliates (unless you want to do that?).
As far as external links go, though, it's typically okay (and expected) that you link out to other sites.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My affiliate links are being crawled by Google. Is this a bad thing? Also, should I add rel: nofollow to affiliate links?
Hi All, I have an issue on Google Search Console. It seems all my affiliate links are being crawled. Is this a bad thing? The reason I found out is that all but one link was flagged with a warming. It goes on to say that it was blocked by robots.txt. The strange thing is that I've not set any links to be blocked in robots.txt. Also, should affiliate links be nofollow? I'm asking because I was recently sent this article by Yoast on how to properly cloak my affiliate links without using a plugin. https://yoast.com/cloak-affiliate-links/ In the method, they apply the nofollow and noindex rules to the links. I'm just a little confused on how to go about affiliate links the SEO friendly way.
Affiliate Marketing | | nomad_blogger0 -
NoFollow Affiliate links from (very) shady websites will effect rankings?
Should I be worried , and if so what to do, if affiliates are linking from shady / gaming/adult websites? or dost the fact that the links are nofollow solve this issue completely ?
Affiliate Marketing | | KSafer0 -
Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
Hey all, I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something? Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis. Bi75jku
Affiliate Marketing | | BradsDeals0 -
Which is the Best Affiliate Marketing Platform in Australia?
We have 1 eCommerce website (www.bannerbuzz.com.au) for printing products like banners, signs, banner stands, teardrop flags and more... We want to start affiliate marketing for our products with CPA model. So, Do anyone have a good experience with Australian affiliate marketing platform. We are having shareasale account for our USA based website, But Shareasale is not that much aggressive in Australia. So we want to finalize 1 platform for affiliate marketing which can have good strength of advertisers and reputation. Thanks,
Affiliate Marketing | | CommercePundit0 -
'legitimate' link wheels
I was wondering what SEOMoz' thoughts are on the mega legitimate link wheel sites that are out there. TechMediaNet have been buying up massive news/media sites which arent really monetised (adsense) like http://www.space.com/ http://www.ouramazingplanet.com/ livescience.com and others and generating, admittedly good quality, curated content. Then seeding them with content with backlinks to their money site, toptenreviews.com which in essence is a review site with thousands of pages loaded with affiliate links no better than any of the other site out there. e.g. http://www.livescience.com/9755-bing.html due to the scale of what they're doing pretty much any keyword search i do with review in it (the last thing i wanted to purchase was a usb 3 hub) ends up with toptenreviews.com dominating the serps presumably due to the high PR the viral nature of the media sites are working with. Do you think Google will crack down on this, or do you think it's capitalism in action? TechMediaNet have invested millions in this project and just gotten another $33 million in funding recently.....
Affiliate Marketing | | idimmu0 -
Can linking out to a weak site harm my sites SEO?
We have an affiliate that wants us to link to his site to help him get started. His website looks okay and is in the same niche as us. However, he has PR 0 and no Page Trust. We could work in a natural looking link in a news article, but there is no way he could be considered any kind of "authority" link. Can linking to his site harm us?
Affiliate Marketing | | theLotter0 -
Affiliate links and Panda
Hello all, my question relates to the recent panda changes and the subsequent impact on pages with affiliate links. I have a hub of pages on my website talking about parcel delivery. Each page has unique, quality content and up until about 3 weeks ago they were starting to achieve strong rankings. We then decided to add some affiliate links (just 3) on each of the 5 parcel related pages we have. The rankings for each of these pages have tanked completely; rather than top 20, we're now outside of the top 100 for all of them. When we noticed this we masked the link, changing it to domain.com/index/affiliate; we also no followed the link. Alas this has had minimal effect in restoring the rankings. What i'm now thinking of doing is removing the affiliate links from the parcel landing pages altogether and instead providing users with a no followed link to another page within the site upon which we would have the affiliate links. This page would also be no indexed as well. My question is: Is this a good idea and if so will it restore my previous rankings over time, or have those pages (as a result of the affiliate links) been permanently down graded?? Any assistance would be gratefully received!
Affiliate Marketing | | danielparry0 -
Does an affiliate link bring the same SEO juice as a standard link?
I wonder if affiliate links, like the ones offered by Amazon Associates program bring the same SEO as would a link to the same page without this additional "ref=..."?
Affiliate Marketing | | maciek-0