Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html) -
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Find archived sitemap of a website that no longer exists
I am trying to figure out the site structure of a website and the urls of all the pages. Normally this would be easy but a couple of months ago the website went down and I don't think it will ever come back. Any help would be appreciated.
Competitive Research | | EdKim1 -
What to do about one site dominating search results? (multiple pages ranking)?
Anybody have thoughts on dealing with search results where the same site gets listed multiple times? "weebly vs wix" is one example (same site #1-3, repetitive articles, not crazy high authority), but I see this now and then. I know Google likes variety, so it's weird for me to see results like this dominating search results. Thoughts? What gets these sites to take over the top rankings for a specific term? Any way to rise up in this situation, outside of the usual? Any tips on duplicating this kind of success?
Competitive Research | | davidwaring0 -
I am looking to find the top pages based on traffic volume on my competitors websites, does anyone know of any good resources?
I want to know how which pages on my competitors websites are the most popular based on the traffic volume. I do not care how many links or directed to that page or any other metric. Only thing I am looking for is the traffic volume. It would also be nice to know the length of time spent on that page.
Competitive Research | | kanteenboy0 -
Why do some sites perform great but show no data on the tool bar?
Hey everyone, Been a paid member for 3 days and this is my first time posting 🙂 I have a quick question about competitive analysis with the MozBar. I do not want to point out the exact website (unless that is something cool to do on this forum), but there is a site ranking for a single keyword in the local 7 pack in slot one, which is also slot one for the SERP entirely. When I go to look at the page in more detail, the MozBar is showing no DA, PA, anything at all, and the Open Site Explorer is showing no links internal or external. I know that the page has to have at least some of these elements populated in order to be ranking for a term this competitive, so why is the bar not showing anything? The bar works GREAT on other sites below this one. Thanks for any help!
Competitive Research | | tonyflorespsa0 -
How Do You Create an excel spreadsheet of all blog post content on a site?
Is there a quick and easy way to create an excel spreadsheet with a list of all blog content and it's SEO factors i.e. URL, title, description, etc.? I know I could use screaming from to get the entire site but is there a way to just get the content on the blog?
Competitive Research | | RonMedlin0 -
Can I write guest blogs on competitor's blogs?
I have been doing some guest blogging and its hard to find guest blogs opportunities. I have found some but they are direct competitors. Is it safe to write guest post? by the way the are not local competitors. Any suggestion please. thanks
Competitive Research | | conversiontactics0 -
How can I obtain analytics and social data for a site that I'm not authorized on?
I'd like to get traffic information and facebook /twitter stats for a company. Don't know their ua, logins or have access to any of their pages. Where can I find this information? Any help would be greatly appreciated!
Competitive Research | | brandonschultz0 -
Does it makes any difference if i use 's' at the end of keyword.
In terms of ranking and other stuff, how does adding 's' at the end of a keyword works ? For example - 'Custom Cases' in place of 'Custom Case' I am getting different results as well for both. Please advice....
Competitive Research | | viniyog0