Adding the link masking directory to robots.txt?
-
Hey guys,
Just want to know if you have any experience with this.
Is it worthwhile blocking search engines from following the link masking directory.. (what i mean by this is the directory that holds the link redirectors to an affiliate site:
example:
mydomain.com/go/thislinkgoes to
amazon.com/affiliatelinkI want to know if blocking the 'go' directory from getting crawled in robots.txt is a good idea or a bad idea?
I am not using wordpress but rather a custom built php site where i need to manually decide on these things.
i want to specifically know if this in any way violates guidelines for google. it doesn't change the custom experience because they know exactly where they will end up if they click on the link.
any advice would be much appreciated.
-
Iredeto,
I do this on a few of my sites and it works out well. Saves on crawl budget, keeps google from accessing my affiliate links, and keeps any pagerank from passing through the links, which keeps me in-line with Google's webmaster policy on links.
One thing to keep in mind is that Google may show those URLs in the search results with a message that they can't show the content because it has been blocked. If you want to keep this from happening you may have to remove that directory GWT using the URL Removal tool. If they get re-indexed in 90 days (or whatever the reset time frame is) you will have to do it again. Hopefully that won't be an issue once you get them all removed the first time and block the folder.
Using rel = "nofollow" tags on the hyperlinks going to that directory wouldn't hurt either.
-
Hi!
I know cases where you can hide the contents of a folder to Google to not affect the rest of the website.
Note that you are facing a machine that knows how to crawl and index content.
Also be sure to block with noindex nofollow meta tags and removing URLs from Google index if at some point you had not locked.
Best regards!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Multiple links from same domain (different pages) considered in credibility of backlinks?
Hi, Let's say there are multiple backlinks from different pages of same domain to different pages of other domain like below: Website A: Page 1 -----------> Website B: Page 1 Website A: Page 2 -----------> Website B: Page 2 Do the pages of Website B pages will get backlinks authority equally or they don't get much backlinks impact as they have multiple backlinks from same domain? There were old school stories that Google ignores second link from same domain.....etc... So, please suggest on this. Thank you. Note: The question is NOT about content relevancy or domain authority score of the backlinks.
Algorithm Updates | | vtmoz1 -
Do the back-links go wasted when anchor text or context content doesn't match with page content?
Hi Community, I have seen number of back-links where the content in that link is not matching with page content. Like page A linking to page B, but content is not really relevant beside brand name. Like page with "vertigo tiles" linked to page about "vertigo paints" where "vertigo" is brand name. Will these kind of back-links completely get wasted? I have also found some broken links which I'm planning to redirect to existing pages just to reclaim the back-links even though the content relevancy is not much beside brand name. Are these back-links are beneficial or not? Thanks
Algorithm Updates | | vtmoz0 -
Too many "nofollow" outgoing links are Okay?
Hi all, Our forum have so many discussions and topics where our users leave their websites and oter URLs which will be marked "nofollow" by default. Beside spammy websites, is that Okay to have so many "nofollow" outgoing links? Thanks
Algorithm Updates | | vtmoz0 -
Relevant Link, but Low DA...good idea?
If a website has a low DA (not because of spam. Just because it's new or because there isn't a ton of content) but it is industry specific/relevant, then is that worth pursuing? I have read how relevancy is supposed to be a major portion determining a link's benefit, but I"m leery about about something with a low DA - like under 15 low. Thanks, Ruben
Algorithm Updates | | KempRugeLawGroup0 -
Ecommerce SEO: Is it bad to link to product/category pages directly from content pages?
Hi ! In Moz' Whiteboard friday video Headline Writing and Title Tag SEO in a Clickbait World, Rand is talking about (among other things) best practices related to linking between search, clickbait and conversion pages. For a client of ours, a cosmetics and make-up retailer, we are planning to build content pages around related keywords, for example video, pictures and text about make-up and fashion in order to best target and capture search traffic related to make-up that is prevalent earlier in the costumer journey. Among other things, we plan to use these content pages to link directly to some of the products. For example a content piece about how to achieve full lashes will to link to particular mascaras and/or the mascara category) Things is, in the Whiteboard video Rand Says:
Algorithm Updates | | Inevo
_"..So your click-bait piece, a lot of times with click-bait pieces they're going to perform worse if you go over and try and link directly to your conversion page, because it looks like you're trying to sell people something. That's not what plays on Facebook, on Twitter, on social media in general. What plays is, "Hey, this is just entertainment, and I can just visit this piece and it's fun and funny and interesting." _ Does this mean linking directly to products pages (or category pages) from content pages is bad? Will Google think that, since we are also trying to sell something with the same piece of content, we do not deserve to rank that well on the content, and won't be considered that relevant for a search query where people are looking for make-up tips and make-up guides? Also.. is there any difference between linking from content to categories vs. products? ..I mean, a category page is not a conversion page the same way a products page is. Looking forward to your answers 🙂0 -
Googlebot soon to be executing javascript - Should I change my robots.txt?
This question came to mind as I was pursuing an unrelated issue and reviewing a site's robots/txt file. Currently this is a line item in the file: Disallow: https://* According to a recent post in the Google Webmasters Central Blog: [http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better") Googlebot is getting much closer to being able to properly render javascript. Pardon some ignorance on my part because I am not a developer, but wouldn't this require Googlebot be able to execute javascript? If so, I am concerned that disallowing Googlebot from the https:// versions of our pages could interfere with crawling and indexation because as soon as an end-user clicks the "checkout" button on our view cart page, everything on the site flips to https:// - If this were disallowed then would Googlebot stop crawling at that point and simply leave because all pages were now https:// ??? Or am I just waaayyyy over thinking it?...wouldn't be the first time! Thanks all! [](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better")
Algorithm Updates | | danatanseo0 -
Paid for Directory Listings and no-follows
Was just about to book an advert in a directory until the salesperson told me that the directory listing would be replicated over 40 times, from different websites (they white label the directory to different sites, who then sell their own adverts in the directory). I asked them whether they did nofollows - they didn't know what they were, so clearly not. I'm thinking 40 plus identical links coming through (over 10% of total links to site) could attract punishment from Google. Your thoughts would be welcome. Thanking you in advance, Luke
Algorithm Updates | | McTaggart0 -
Traffic dropping after blog was added...
We have a major client that gets over a thousand organic visits per day. They also self manage an adwords campaign that generates a few thousand clicks a day. Lat summer they got a new website with a blog. The blog has a couple of teaser paragraphs on the home page with the new blog posts, in addition to lots of static text about the items sold on the site. What has happened is, our organic traffic has changed form the main keywords of what the site sells, to things talked about in the blog. Traffic hasn’t dropped dramatically, but we have lost all our big keywords, for example, a keyword that was generating 40 organic clicks per day a year ago we receive no traffic for now, and much more obscure keyword combinations from the blog are being picked up. Is it the addition of the blog that is causing this problem, or the blog teasers that are on the home page of the site? We like the traffic from the blog terms, but we would like our former main keyword traffic back as well, how can we accomplish this, and what do you think has occurred here? Thanks for your help.
Algorithm Updates | | WhiteHat120