Restricted by robots.txt does this cause problems?
-
I have restricted around 1,500 links which are links to retailers website and links that affiliate links accorsing to webmaster tools
Is this the right approach as I thought it would affect the link juice? or should I take the no follow out of the restricted by robots.txt file
-
Hello Ocelot,
I am assuming you have a site that has affiliate links and you want to keep Google from crawling those affiliate links. If I am wrong, please let me know. Going forward with that assumption then...
That is one way to do it. So perhaps you first send all of those links through a redirect via a folder called /out/ or /links/ or whatever, and you have blocked that folder in the robots.txt file. Correct? If so, this is how many affiliate sites handle the situation.
I would not rely on rel nofollow alone, though I would use that in addition to the robots.txt block.
There are many other ways to handle this. For instance, you could make all affilaite links javascript links instead of href links. Then you could put the javascript into a folder called /js/ or something like that, and block that in the robots.txt file. This works less and less now that Google Preview Bot seems to be ignoring the disallow statement in those situations.
You could make it all the same URL with a unique identifyer of some sort that tells your database where to redirect the click. For example:
www.yoursite.com/outlink/mylink#123
or
www.yoursite.com/mylink?link-id=123
In which case you could then block /mylink in the robots.txt file and tell Google to ignore the link-ID parameter via Webmaster Tools.
As you can see, there is more than one way to skin this cat. The problem is always going to be doing it without looking like you're trying to "fool" Google - because they WILL catch up with any tactic like that eventually.
Good luck!
Everett
-
From a coding perspective, applying the nofollow to the links is the best way to go.
With the robots.txt file, only the top tier search engines respect the information contained within, so lesser known bots or spammers might check your robots.txt file to see what you don't want listed, and that info will give them a starting point to look deeper into your site.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt error
Moz Crawler is not able to access the robots.txt due to server error. Please advice on how to tackle the server error.
Technical SEO | | Shanidel0 -
Location-Aware Browsing issue. Could it cause a drop in rankings
Hello Mozzers. A client has seen big drop in their organic rankings in google.co.uk. Desktop results have dropped from page one to page three and mobile results have completely gone. They have recently added Location-Aware Browsing to the site so that it attempts to find you location and add this to the search box. However this appears to be in a loop and doesn't then turn off once you have allowed it. It keeps appearing in both Firefox and Chrome and asking for permission to use you location. Could this have resulted in the ranking drop? There have also rolled out a new mobile version and this has the same issue. Any ideas? Thanks in advance TH
Technical SEO | | highwayfive0 -
Problem of possible duplicate title tag and description. Help me!
Hi everybody, I'm optimizing this huge website that has a lot of identical categories for differente locations. I'm trying to find a smart way to write title and description for these categories, changing the location as a variable on the title and description phrase. Here some examples: Title: Attractions in [CITY]. Sightseeings, monuments and museums in [CITY]. Description: Find travel ideas and suggestions for [CITY]. On [NAME OF THE WEBSITE] you can find a lot of attractions, monuments and sightseeing off the beaten path in [CITY]. Changing only the name of the CITY on these Titles and Descriptions, am I running the risk of duplicate title and description? Thanks in advance for your help!
Technical SEO | | OptimizedGroup0 -
Problem indexing web developed with Ruby on Rails
Hi there! Here we are again, we are having problems indexing one of our clients, which website has been developed with Ruby on Rails. It doesnt get the titles right from almost all our pages...Has anyone had the same problem? Any feedback would help a lot... Thanks!
Technical SEO | | Comunicare0 -
Is anyone having problems with sending emails
I have been having problems for a number of weeks now, where if i send a couple of emails out then all of a sudden i am blocked and have to ask my hosting company for a new ip address. my site is in joomla and my email address is through my site. the messages i am getting when sending emails is as follows, and this stays like this until i get a new ip address A message that you sent could not be delivered to one or more of its
Technical SEO | | ClaireH-184886
recipients. This is a permanent error. The following address(es) failed: 550-5.7.1 [184.154.89.211 1] Our system has detected an unusual rate of
550-5.7.1 unsolicited mail originating from your IP address. To protect our
550-5.7.1 users from spam, mail sent from your IP address has been blocked.
550-5.7.1 Please visit http://www.google.com/mail/help/bulk_mail.html to review
550 5.7.1 our Bulk Email Senders Guidelines. r2si12781844igh.70 - gsmtp any help would be great0 -
How to solve the meta : A description for this result is not available because this site's robots.txt. ?
Hi, I have many URL for commercialization that redirects 301 to an actual page of my companies' site. My URL provider say that the load for those request by bots are too much, they put robots text on the redirection server ! Strange or not? Now I have a this META description on all my URL captains that redirect 301 : A description for this result is not available because this site's robots.txt. If you have the perfect solutions could you share it with me ? Thank You.
Technical SEO | | Vale70 -
Clone TLD Problems
I have an online services website www.geekwik.com which I started 3 months back. I also recently made a clone in TLD geekwik.in which has the same content, only pricing is in INR and is targeted at India users, while geekwik.com is targetted at global users with pricing in USD. How do I manage these 2 sites so that I do not face duplicate content penalty from google and the sites do not cannibalize on each other. Is there anything specific I need to do in robots.txt or .htaccess or sitemaps or hrelang etc? I personally feel that after putting up geekwik.in couple of weeks ago, the ranking of geekwik.com went down and I started getting lesser search queries. I would be putting up an IP based switch on both sites shortly so that Indian users are redirected to .in TLD and non-Indians are redirected to .com TLD. From SEO standpoint what are the things I need to do to counter these problems mentioned above. Putting India version in a subdirectory is also an option.
Technical SEO | | geekwik0 -
Same URL in "Duplicate Content" and "Blocked by robots.txt"?
How can the same URL show up in Seomoz Crawl Diagnostics "Most common errors and warnings" in both the "Duplicate Content"-list and the "Blocked by robots.txt"-list? Shouldnt the latter exclude it from the first list?
Technical SEO | | alsvik0