Adding the link masking directory to robots.txt?
-
Hey guys,
Just want to know if you have any experience with this.
Is it worthwhile blocking search engines from following the link masking directory.. (what i mean by this is the directory that holds the link redirectors to an affiliate site:
example:
mydomain.com/go/thislinkgoes to
amazon.com/affiliatelinkI want to know if blocking the 'go' directory from getting crawled in robots.txt is a good idea or a bad idea?
I am not using wordpress but rather a custom built php site where i need to manually decide on these things.
i want to specifically know if this in any way violates guidelines for google. it doesn't change the custom experience because they know exactly where they will end up if they click on the link.
any advice would be much appreciated.
-
Iredeto,
I do this on a few of my sites and it works out well. Saves on crawl budget, keeps google from accessing my affiliate links, and keeps any pagerank from passing through the links, which keeps me in-line with Google's webmaster policy on links.
One thing to keep in mind is that Google may show those URLs in the search results with a message that they can't show the content because it has been blocked. If you want to keep this from happening you may have to remove that directory GWT using the URL Removal tool. If they get re-indexed in 90 days (or whatever the reset time frame is) you will have to do it again. Hopefully that won't be an issue once you get them all removed the first time and block the folder.
Using rel = "nofollow" tags on the hyperlinks going to that directory wouldn't hurt either.
-
Hi!
I know cases where you can hide the contents of a folder to Google to not affect the rest of the website.
Note that you are facing a machine that knows how to crawl and index content.
Also be sure to block with noindex nofollow meta tags and removing URLs from Google index if at some point you had not locked.
Best regards!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do orphan pages take away link juice?
Hi, Just wondering about this whether the orphan pages take away any link juice? We been creating lot of them these days only to link from external sites as landing pages on our site. So, not linking from any part of our website; just linking from other websites. Also, will they get any link juice if they are linked from our own blog-post? Thanks
Algorithm Updates | | vtmoz1 -
What happens if we remove all the links to internal pages from our homepage?
Hi Moz community, We wanna give a try by removing all the links from homepage to internal pages and keep just a free trial button. Will this impact our SEO anyway? We have nearly 15 important internal pages at 2nd and 3rd hierarchy level. They may drop in rankings but we want to risk for few days to understand how it works. Your opinion please! Thanks
Algorithm Updates | | vtmoz0 -
Homepage title tag: "Keywords for robots" vs "Phrases for users"
Hi all, We keep on listening and going through the articles that "Google is all about user" and people suggesting to just think about users but not search engine bots. I have gone through the title tags of all our competitors websites. Almost everybody directly targeted primary and secondary keywords and few more even. We have written a very good phrase as definite title tag for users beginning with keyword. But we are not getting ranked well comparing to the less optimised or backlinked websites. Two things here to mention is our title tag is almost 2 years old. Title tag begins with secondary keyword with primary keyword like "seo google" is secondary keyword and "seo" is primary keyword". Do I need to completely focus on only primary keyword to rank for it? Thanks
Algorithm Updates | | vtmoz0 -
Would you "nofollow" links from a column on HuffingtonPost?
Hi all, So, I've read a lot of posts about guest posting being dead, but what about if you have a regular column on a well-regarded site? Stop? Nofollow links? We have a regular column on the Huffington Post and each piece has historically had at least one link (or more) back to our site. Yes, early on (like last year) we did use optimized anchor text in our links, and then calmed down on that a bit. But regardless, the links have always been relevant to the topic covered, and the topic is always in our niche (namely: budget travel in Europe). I saw Matt Cutts' recent video in which he recommends using the "nofollow" tag on guest posts when linking to one's own site, and specifically mentions HuffPo. Thus, I'm prepared to go back to my old posts and "nofollow" those links, but I just wanted a sanity check from the fine folks at SEOMoz. Would you go back and nofollow them? Many thanks!
Algorithm Updates | | TomNYC0 -
Does adding lots of new content on a site at one time actually hurt you?
When speaking with a client today, he made the comment that he didn't want all of the new content we'd been working to be added to the site all at once for fear that he would get penalized for flooding the site with new content. I don't have any strong data to confirm or refute the claim, is there any truth to it?
Algorithm Updates | | JordanRussell0 -
So, useless link exchange pages still work?!
After 3 years out of SEO I thought things might have moved on, but apparently not. Bit of back link research and all the top sites in my niche have tons of reciprocal links to barely relevant sites. Do I really have to do this? I mean I thought this was so out of date, it's not much better than keyword stuffing. So, should I just forget my lofty principles asking myself 'is this of any value to my users?' and just take the medicine?
Algorithm Updates | | Cornwall0 -
Linking to authority / competitor website!? Yes or No?
Here is the dilema .... People say, don't be afraid to link to authority website! What if an authority website in particular niche is actually competing with you in the Search, but you do have a review profile there? Is it ok to link to that profile ? Should I include a no follow tag ? Another case : some other authority high ranked website competing in search with mine directly has a profile page for my company, but this authority website has blocked Google bot to crawl the profile page all the reviews there !? Can I still link to that page and will this be appreciated by Google ? Am I passing PR and link juice from my website to those direct comeptitors / authority websites ?
Algorithm Updates | | montauto0 -
Redirected old domain to new, how long before seeing the external links under the new domain?
Before contracting SEO services, my client decided to change his established root domain to one more customer-friendly. Since he had no expertise on board, no redirects were set up until 6 months later. I ran stats right before the old domain was redirected and have a report showing that he had roughly 750 external links from 300 root domains. We redirected the old domain to the new domain in mid Jan 2012. Those external links are still not showing in Open Site Explorer for the new domain. I've tested it a dozen times, and the old domain definitely points to the new domain. How long should it take before the new domain picks up those external links? Should I do anything else to help the process along?
Algorithm Updates | | smsinc0