How to Find Another Site's robots.txt File?

[lbohen]

An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.

How do I find out what is in their robots.txt files?

[lbohen]

Should I disallow 80legs and sitebot robots?

Why might these two robots be disallowed?

[riyas_]

1. audiobooks.com

User-agent: 008
Disallow: /

(Tells 80legs Robot to stay out of the website)

User-agent:*
Disallow:
```

(Tells all other robots to visit all files of the website)

  2) audible.com

```
User-agent: sitebot
disallow: /
```

(Tells Sitebot Robot to stay out of the website)

```
User-agent: *
Disallow: /mycart
Disallow: /ajaxcart
Disallow: /create-account
Disallow: /acc-merge
Disallow: /acc-merge6for6
etc.. 
```

(Tells all other robots not to enter specific directories listed)

Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)

[lbohen]

I would appreciate help in understanding what the following robots.txt files are doing.

http://www.audiobooks.com/robots.txt

http://www.audible.com/robots.txt

[riyas_]

Just go to example.com/robots.txt

eg: http://www.facebook.com/robots.txt , http://www.seomoz.org/robots.txt

[PaulRonin]

Hey Larry,  You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt