Adding the link masking directory to robots.txt?
-
Hey guys,
Just want to know if you have any experience with this.
Is it worthwhile blocking search engines from following the link masking directory.. (what i mean by this is the directory that holds the link redirectors to an affiliate site:
example:
mydomain.com/go/thislinkgoes to
amazon.com/affiliatelinkI want to know if blocking the 'go' directory from getting crawled in robots.txt is a good idea or a bad idea?
I am not using wordpress but rather a custom built php site where i need to manually decide on these things.
i want to specifically know if this in any way violates guidelines for google. it doesn't change the custom experience because they know exactly where they will end up if they click on the link.
any advice would be much appreciated.
-
Iredeto,
I do this on a few of my sites and it works out well. Saves on crawl budget, keeps google from accessing my affiliate links, and keeps any pagerank from passing through the links, which keeps me in-line with Google's webmaster policy on links.
One thing to keep in mind is that Google may show those URLs in the search results with a message that they can't show the content because it has been blocked. If you want to keep this from happening you may have to remove that directory GWT using the URL Removal tool. If they get re-indexed in 90 days (or whatever the reset time frame is) you will have to do it again. Hopefully that won't be an issue once you get them all removed the first time and block the folder.
Using rel = "nofollow" tags on the hyperlinks going to that directory wouldn't hurt either.
-
Hi!
I know cases where you can hide the contents of a folder to Google to not affect the rest of the website.
Note that you are facing a machine that knows how to crawl and index content.
Also be sure to block with noindex nofollow meta tags and removing URLs from Google index if at some point you had not locked.
Best regards!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Canonicals from sub-domain to main domain: How much content relevancy matters? Any back-links impact?
Hi Moz community, I have this different scenario of using canonicals to solve the duplicate content issue in our site. Our subdomain and main domain have similar landing pages of same topics with content relevancy about 50% to 70%. Both pages will be in SERP and confusing users; possibly search engine too. We would like solve this by using canonicals on subdomain pointing to main domain pages. Even our intention is to only to show main domain pages in SERP. I wonder how Google handles it? Will the canonicals will be respected with this content relevancy? What happens if they don't respect? Just ignore or penalise for trying to do this? Thanks
Algorithm Updates | | vtmoz0 -
Domain has been redirected our site; but many incoming links from sub domain. Will this hurts?
Hi all, This is the scenario: Our website is newwebsite.com. Our old website is oldwebsite.com which has been redirected to newwebsite.com (years back). But one of the old website's sub domain has a lot of back links to our current website like: seo.oldwebsite.com to newwebsite.com. Will this scenario hurts with any wrong linking? Thanks
Algorithm Updates | | vtmoz0 -
Drastic Drop in Link Juice
Hi Back in December we shifted my web domain from a gourmetdirect.com to gourmetdirect.co.nz as part of a site-wide revamp. Everything was going along fine until recently when my Linking domains plummeted and external links fell from 6000 approx to 600. We still have the .com live for loads of disfunctional reasons. Can anyone help? I have gone from a top ranker to a no show and my contractors are all shaking their heads.
Algorithm Updates | | GourmetDirect0 -
Links to category pages unnatural?
If people are linking to your site, it would seem natural that the vast majority of those links would point to the homepage, product page, or a article/content page. Let's say you have 100 links pointing to your site, and 40 of them are pointing to category pages. Would this seem unnatural? Does Google or other search engines have a way of determining this as a factor in ascertaining whether the links are natural or not? Is there a rule of thumb when it comes to the pages that are linked to on your site?
Algorithm Updates | | inhouseseo0 -
Direct Domain Name Anchor Text Spammy Links
Hello! I have a website that has been hit with around 120-150 spammy bookmarking sites which I believe are just scraping content from one another or were added by someone that was hired earlier or maybe some other action, but that really doesn't matter. My question is whether I should be worried about that many domains linking to the site in question with anchor text that is "www.domainname.com" and linking to the domain itself? I have done quite a few researches on this issue and the general conclusion is these don't help, but they don't hurt your rankings either. I wanted to hear from the SEOMoz community about it though. My opinion is Google doesn't take them seriously and we shouldn't worry about them, try to take them off and we should simply work on our content, guest posts, produce our generally great deals on our services and move on. Thanks!
Algorithm Updates | | Njave_MCP0 -
Best way of seeing how many links come from individual root domain domains.
Just wondering how best to see this - which tool to use. I'm dealing with a website with several thousand inbound links from around 100 root domains. Thanks in advance, Luke
Algorithm Updates | | McTaggart0 -
Difference between Google's link: operator and GWT's links to your sites
I haven't used the Google operator link: for a while, and I noticed that there is a big disparity between the operator "link:" and the GWT's links to your site. I compared these results on a number of websites, my own and competitors, and the difference seem to be the same across the board. Has Google made a recent change with how they display link results via the operator? Could this be an indication that they are clean out backlinks?
Algorithm Updates | | tdawson090 -
Google Directory vs DMOZ
What is the difference between the Google Directory and the DMOZ if any?
Algorithm Updates | | BrandonC-2698870