Best way to block a search engine from crawling a link?

[nicole.healthline]

If we have one page on our site that is is only linked to by one other page, what is the best way to block crawler access to that page?

I know we could set the link to "nofollow" and that would prevent the crawler from passing any authority, and we can set the page to "noindex" to prevent it from appearing in search results, but what is the best way to prevent the crawler from accessing that one link?

[deltasystems]

Hi there,

I'm assuming you are trying to do pagerank sculpting (or something related..) - which was made a little more tough in recent years. I'll base my answer around this assumption, so feel free to correct me if this isn't the case.

There are several methods to make a link uncrawlable:

1. AJAX - Googlebot will not read any calls through AJAX. If you can load your link through an external call, it would be completely hidden.
2. Javascript - Obfuscate links with Javascript that masks the link. You can do any number of solutions here, including using tags with a title of your URL, which upon clicking, goes that that URL. Simple and effective.
3. Redirects - I haven't tested this last idea, and it may not work. You might be able to redirect to another page in your website, which is then set to not be indexed. Then redirect to the intended page through a query string. In theory it should work, but obviously not as good as the previous methods I described.

Let me know if you have questions. I'd be glad to help further.

Cheers!

[Nick_Ker]

Noindex/nofollow should be good enough, but if you want to be sure it doesn't get indexed, you could can also include <meta name="robots" content="NOINDEX, NOFOLLOW"> in the head section of the page to be blocked. You can also exclude the page in your robots.txt file. </meta name="robots">

You can find a simple robots.txt generator in Google Webmaster Tools if you need to block particular pages or directories. The robots.txt file should be in the root directory of your site and look something like this:

User-agent: * Disallow: /file-you-want-to-hide.html

You can also request removal of specific URLs in Webmaster Tools if it has already been indexed.