Blocking external links in Robots.txt - need advice on Best Practice
-
I look after an affiliate site that is doing quite well in the search rankings. We've been doing a review of our practices and one thing that has cropped up is our robots.txt.
In it, we disallow Google from crawling external links. This used to be best practice in the affiliate industry a couple of years ago, but I wanted to know if this is still the case, and what the implications are if we were to:
a/ leave it as is?
b/ allow crawling?
Thanks in advance.
-
I'm not sure how you're blocking Google from crawling external links in the robots.txt file--typically you only block them from crawling internal pages on your site.
If you're using a script, though, to track the clicks on external links and that script is running your site (and you're blocking that script in robot.txt), then that still should be fine. You may want to add a "nofollow" tag on those links, though, so you don't end up passing link "credit" or "link juice" to those affiliates (unless you want to do that?).
As far as external links go, though, it's typically okay (and expected) that you link out to other sites.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Low value link building to sitemap.xml
During some competitive research recently I discovered one of my clients competitors sites had an interesting backlink profile. Looking at the top-pages report in Open Site Explorer the home page was the #1 page (as you'd expect) with 2.5k links from about 500 linking root domains The second page was the sitemap.xml (~1.5k links, 400 linking root domains) and the third was their /feed page (again, ~1.5k links, 350 linking root domains). Links to these two pages aren't something that would happen naturally (particularly the sitemap.xml). There's a whole load of evidence for nasty low quality link building such as over-optimised keyword rich anchor text, comment spam, and even some blog/article based link networks. It's a pretty nasty niche with lots of cut-throat affiliate marketing. My guess here is that someone may have made a mistake using an automated link building too, but I'd be interested in what you might think? Have you seen this before? (Sorry, I can't reveal the domains in question as I'm bound by an NDA.)
Affiliate Marketing | | DougRoberts0 -
Choosing an Affiliate Software & Link SEO-Value
Hello everyone. I'm looking for recommendations for affiliate software. I'm also curious if there is any truth in the "SEO-value" that some softwares are touting as a benefit. Choosing a Software: This Moz Blog Article says to build it yourself. However, I'm inclined to use an out-of-the-box software for advanced features such as real-time analytics for the affiliate, w-9/1099 processing, etc. MozPerks promotes hasoffers.com which seems overly pricey ($230/month with perk discount) and OMNISTAR who just left me on hold for 10 minutes (and still no anwer from their sales department). So I'm still looking. Any recommendations? SEO-value of Links from Affiliates: Here's an example of what OMNISTAR claims about the SEO value of the way they structure the links from affiliate sites to my site. I hear a variation on this pitch from each software and none of them seem to agree. Does Google want me to clearly tag promotional vs. editorial links? Is there SEO-value in promotional links on affiliate sites or is that a load of bologna? Thanks a bunch!
Affiliate Marketing | | Harbor_Compliance0 -
Find all affiliate links in a custom CMS
I have a custom CMS. I offer a few affiliate products through my site. Now I need to update all my aff. links and I'm not sure how I can find all the pages (posts) on my site that have affiliate links. What's the best way to do that?
Affiliate Marketing | | VinceWicks0 -
What is the negative effect of getting affiliate links to your domain?
Are the negative effects of affiliate links to your domain bigger than the positive effects? Is it better to use a subdomain for your affiliate campaigns? (Because of linkprofile no follow links or time on page effects?) Thanks for responce!
Affiliate Marketing | | ltom0 -
Sponsored Links on Amazon, help
So from what I can see CLickrive was discontinued by Amazon, replaced by a9. I have been asked by a client how they can get onto Sponsored Links near the bottom of a search query on an amazon page. Now I can see from the amazon associates site that you can put widgets, display ads, product search links etc but there is no mention of 'Sponsored Links' other than: SPONSORED LINKS Sponsored links are advertisements related to your recent product search query or content on the page. Sponsored links are always clearly labeled. When you click on a sponsored link, we get revenue. Generating additional revenue from sponsored links allows Amazon.com to offer lower prices to you - something we are dedicated to doing in every way we can. There is no mention on how to actually be a sponsored link... I have read on various blogs etc that it is delivered by a third party network... no mention as to whom... thought Google but no mention of this on any information I can find on Google blogs etc. Anyone have an idea? Thank you 🙂
Affiliate Marketing | | ChalkWard0 -
Can anyone help me find the broadband ISP affiliate program I need?
I have a couple sites that deal with broadband internet access, and I'm looking to monetize beyond adsense. I'd like to sign up for some affiliate programs from internet service providers like Charter, CenturyLink, Comcast, etc. I found a couple affiliate programs but I am having no luck at all figuring out the programs that so many other websites are involved with. In about every case, the user is able to enter in their address and then a search is performed, returning programs available at their location. The programs I've found just do the traditional ad click-throughs for a possible commission. Can anyone enlighten me what kind of programs they are using?
Affiliate Marketing | | bizzer0 -
Big Affiliate Site vs. Small In House Software For Link Value
Hi, I have a client who sells consumer products and is interested in affiliate advertising. Of course, they can get set up with something like CJ.com. They are also interested in possibly using some third party software to take their affiliate effort in house and be able to offer publishers simple urls. The idea is that this might help the link profile for SEO as well. Here are my questions: Does anybody have any experience with the complication level associated with third-party software for link value path? So, relative pluses and minuses. Can anyone recommend an affiiate software that is reliable and easy to use? Any opinion on one big site to go with... CJ, GAN, someone else? Thanks!
Affiliate Marketing | | 945010 -
Affilate links - Google Panda 2.1
With the recent update from Google Panda would links from affiliate Coupon Sites now have an negative impact in Organic Rankings or site value? A reason for this question is because many Coupon Sites usually have poor content and a poor quality score. Thanks
Affiliate Marketing | | Tonyd230