Adding the link masking directory to robots.txt?
-
Hey guys,
Just want to know if you have any experience with this.
Is it worthwhile blocking search engines from following the link masking directory.. (what i mean by this is the directory that holds the link redirectors to an affiliate site:
example:
mydomain.com/go/thislinkgoes to
amazon.com/affiliatelinkI want to know if blocking the 'go' directory from getting crawled in robots.txt is a good idea or a bad idea?
I am not using wordpress but rather a custom built php site where i need to manually decide on these things.
i want to specifically know if this in any way violates guidelines for google. it doesn't change the custom experience because they know exactly where they will end up if they click on the link.
any advice would be much appreciated.
-
Iredeto,
I do this on a few of my sites and it works out well. Saves on crawl budget, keeps google from accessing my affiliate links, and keeps any pagerank from passing through the links, which keeps me in-line with Google's webmaster policy on links.
One thing to keep in mind is that Google may show those URLs in the search results with a message that they can't show the content because it has been blocked. If you want to keep this from happening you may have to remove that directory GWT using the URL Removal tool. If they get re-indexed in 90 days (or whatever the reset time frame is) you will have to do it again. Hopefully that won't be an issue once you get them all removed the first time and block the folder.
Using rel = "nofollow" tags on the hyperlinks going to that directory wouldn't hurt either.
-
Hi!
I know cases where you can hide the contents of a folder to Google to not affect the rest of the website.
Note that you are facing a machine that knows how to crawl and index content.
Also be sure to block with noindex nofollow meta tags and removing URLs from Google index if at some point you had not locked.
Best regards!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ranking drop after sub domain to sub directory migration. Usual?
Hi all, We had our help articles on sub-domain help.website.com. Then we moved it to sub directory website.com/help/. We expected ranking improvement of website.com as there is a wide saying of benefiting from sub domain to sub directory migration. We have noticed that ranking improvement of new sub directory pages (website.com/help/) but not for any main website pages (website.com). I presume that link juice from main website has benefited new sub directory pages but main website lost ranking due to the page rank dilution. Do you agree? Any ideas? Thanks
Algorithm Updates | | vtmoz0 -
Do back-links to non indexed sub-domains / sub-directories considered by Google as website backlinks and pass Pagerank to website?
Hi, If some noindexed links on our website or sub-domain got some backlinks, will that backlinks pass Pagerank / linkjuice to website? Will they be considered as backlinks to website by Google? Here is a statement from Matt cutts for the question. My question is same as below with answer? Eric Enge: Can a NoIndex page accumulate PageRank? Matt Cutts: A NoIndex page can accumulate PageRank, because the links are still followed outwards from a NoIndex page. Thanks
Algorithm Updates | | vtmoz0 -
Footer menu links: Header tags or list items?
Hi, I would like to know header tags (h5 or h6) or list items ( ) works better for footer menu links for the best linking structure. Thanks
Algorithm Updates | | vtmoz1 -
Can we ignore "broken links" without redirecting to "new pages"?
Let's say we have reaplced www.website.com/page1 with www.website.com/page2. Do we need to redirect page1 to page2 even page1 doesn't have any back-links? If it's not a replacement, can we ignore a "lost page"? Many websites loose hundreds of pages periodically. What's Google's stand on this. If a website has replaced or lost hundreds of links without reclaiming old links by redirection, will that hurts?
Algorithm Updates | | vtmoz0 -
How on earth is a site with ONE LINK ranking so well for a competitive keyword?
Ok, so I'm sure you get the gist of what I'm asking about in my question. The query is 'diy kitchens' in Google UK and the website is kitchens4diy[dot]com - which is ranking in third from my viewing. The thing is, the site has just ONE BACKLINK and has done for a good while. Yet, it's ranking really well. What gives?
Algorithm Updates | | Webrevolve0 -
Outsourcing of guest blog articles and usefulness of links from guest blogging
I'm not the greatest writer but want to do some guest blogging for links and traffic. Are there any businesses out there that write world class guest blog articles for subjects that match my business? Also, i've read that doing this for links is really moot because the blog posts get archived and become pagerank "unranked" thereby offering little link value after about a month or so. Once they get archived do they still get counted by google and does the anchor text and page rank still count? Thanks in advance mozzers! Ron
Algorithm Updates | | Ron100 -
One SERP Result, Two Different Link Destinations?
Because my vocabulary isn't up to par, it may be easier for you to skip ahead to the image I've attached. One of my web pages shows up in the Google SERP like this. It has the blue "title" link that goes to one page (URL A), and under that, there is a green "breadcrumb" link that goes to a different page (URL B). Any idea why this is happening and how it can be fixed? Thanks in advance, Benjamin FjhUX.jpg
Algorithm Updates | | 1000Bulbs0 -
Paid for Directory Listings and no-follows
Was just about to book an advert in a directory until the salesperson told me that the directory listing would be replicated over 40 times, from different websites (they white label the directory to different sites, who then sell their own adverts in the directory). I asked them whether they did nofollows - they didn't know what they were, so clearly not. I'm thinking 40 plus identical links coming through (over 10% of total links to site) could attract punishment from Google. Your thoughts would be welcome. Thanking you in advance, Luke
Algorithm Updates | | McTaggart0