How do I disallow crawl on a directory when it's a prefix to my site's URL?

[Simon-Plan]

I am trying to disallow our media repository (hosted elsewhere, but appears as a directory on our site) from being crawled by robots but it is not a subdirectory of the site, it's a prefix.

So I need to disallow: mediabank.mywebsite.org

Not: mysite.org/mediabank

What would I need to put in my robots.txt and/or the other host's robots.txt to make this happen?

Thanks!

[tawnycase]

Hey there! Tawny from Moz's Help Team here.

You'll want to add a robots.txt file for that subdomain, and then add a Disallow command to that robots.txt file. So, using your example, you'd want a file like mediabank.mywebsite.org/robots.txt that had a Disallow command for any robots you don't want crawling that subdomain.

For all user-agents, that would look something like this:

User-agent: *
Disallow: /

That would stop any user-agents from crawling any pages on that subdomain.

I hope this helps! If you've still got questions, feel free to send us a note at help@moz.com and we'll do our best to sort things out for you.

[Alick300]

Hi,

Please check this old thread on the same topic @ https://moz.com/community/q/block-an-entire-subdomain-with-robots-txt

Thanks