Why does SEOMoz crawler ignore robots.txt?
-
The SEOMoz crawler ignores robots.txt
It also "indexes" pages marked as noindex.
That means it is filling up the reports with things that don't matter.
Is there any way to stop it doing that?
-
Hi Alan,
The code should be ok
Try to "drive-test" it with a custom crawl from http://pro.seomoz.org/tools/crawl-test then you will see if it works well.
I am glad the link was useful.
Gr.,
Istvan
-
Thank you István
I added this:
User-agent: rogerbot
Disallow: /sendtoafriend/
Disallow: /photo/
Disallow: /pix/- because crawlers shouldn't go down those paths and roger is detecting pages without descriptions.
Is what I added OK?
-
Hi,
You can block RogerBot from Robots.txt
Check for further instructions on: http://www.seomoz.org/dp/rogerbot
"Please note: Adding this code will prevent our crawl test tool from being able to crawl your website."
Gr.,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Difference between SEOmoz Pro and Mozscape API
Hi, I probably put this question in a wrong category, so I'd like to ask again. I am new in SEOmoz and writing this from Japan.
Moz Pro | | flaminGoGo
I have couple of questions and really appriciate if you can help me to understand SEOmoz. better. First of all, what is the difference between SEOmoz Pro and Mozscape API? I signed up for 30days free trial and now continue being $99 PROmember.
Is that correct that rate limit is one request every 2 seconds with PRO? and also want to know about free API that I found after signed up for PRO.
Is there any restrictions on free API except for rate limit is one request every 10 seconds? The main reason I signed up for SEOmoz was to use the API link metrics, and because of the language I got really confused now.0 -
Is there a way to specify what SEOmoz classes as duplicate content?
Hi all, Currently working through the laundry list of errors and warning on our company's 24 websites. Due to the ridiculous amount of on page links and the sheer volume of products on our sites, much of the descriptive text is similar, following a strict pattern to best mention our USPs and the like. Of course we use a CMS, which means that all the pages look the same and draw this information from the style sheet. Anyways, to the problem at hand. I have been tasked with reducing the "error" count on the SEOmoz admin panel, the problem being SEOmoz is reporting duplicate page content, when they are different, but similar products, for example, 35, 45 and 55 litre refrigeration units. Is there a way in which I can specify what classes as duplicate content, or make the duplicate content report more restrictive, so that everything HAS to be the same for this error to show. Any help is much appreciated, thanks in advance.
Moz Pro | | cmuknbb0 -
Ranking Differs between SEOmoz and Google Webmaster?
My Google webmaster ranking for keywords differs significantly from SEOmoz. For example, Webmaster say 8 and SEOmoz says 1...thoughts?
Moz Pro | | mjcarrjr0 -
SEOmoz Keyword Ranking - Does it work for local search?
I was just wondering how SEOmoz calculates keyword rankings up and down every week. Shouldn't there be a way to have SEOmoz calculate rankings on a Local and Non-Local Basis? For example I have "Local Dentist" as a keyword in SEOmo for my dental client. SEOmoz shows that we are not in the Top 50. | Local dentist Keyword History Ranking Analysis | Not in Top 50 | Moved unknown 0 spaces | Not in Top 50 | Moved unknown 0 spaces | Not in Top 5 | If I change my location in Google browser to the zip of where the office is located, we
Moz Pro | | Czubmeister
show up in position 14(for the home page) & 15 (for an inside page) How can I show the keyword rankings to my client if the keyword ranking report doesn't reflect searches done in my clients local area?0 -
Does SEOmoz have a keyword visibility report / tool?
Know of a tool where I can show the overall success of a set of keywords, us vs. the competition? A visibility report where a #1 ranking is worth 30 points, #2 is worth 29, so on and so on down to #30 worth 1 point, outside of the top 30 is worth nothing. (something like this). Trying to show an overall visibility scorecard, not sure if I can do it here or with some other tool. Didn't see it on Raven. Thanks!
Moz Pro | | akim260 -
Is There a Way to Create Branded Campaign Reports in seomoz.org Campaigns?
We have a few exciting clients we about to start working with and I'm looking for a full campaign management solution that allows for branded reporting. Because we are an agency, is there an option I'm missing that allows you to swap your logo with the seomoz.org logo?
Moz Pro | | stevewiideman0 -
How to set up SEOMOZ to track multilanguage sites?
Hi, I am managing a site in 5 different languages/regions. Our language structure is as follows: www.domain.com/en-us/ - For US www.domain.com/en-uk/ - For UK www.domain.com/fr/ - For France ... However, SEOMOZ does not allow to set up a domain with "/" as a campaign. How can I do it? Thanks
Moz Pro | | hockerty1