Why does SEOMoz crawler ignore robots.txt?
-
The SEOMoz crawler ignores robots.txt
It also "indexes" pages marked as noindex.
That means it is filling up the reports with things that don't matter.
Is there any way to stop it doing that?
-
Hi Alan,
The code should be ok
Try to "drive-test" it with a custom crawl from http://pro.seomoz.org/tools/crawl-test then you will see if it works well.
I am glad the link was useful.
Gr.,
Istvan
-
Thank you István
I added this:
User-agent: rogerbot
Disallow: /sendtoafriend/
Disallow: /photo/
Disallow: /pix/- because crawlers shouldn't go down those paths and roger is detecting pages without descriptions.
Is what I added OK?
-
Hi,
You can block RogerBot from Robots.txt
Check for further instructions on: http://www.seomoz.org/dp/rogerbot
"Please note: Adding this code will prevent our crawl test tool from being able to crawl your website."
Gr.,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What value am I getting from SEOmoz?
Okay, I've been using the trial version of SEOmoz for almost a week, but, I'm just not sure what to do with it to be honest. I'm not an SEO expert, so, a lot of the terminology and reports are confusing. So far the only thing I've found useful are the domain and page duplication errors, but, I can get the same info from Google Webmaster Tools. Am I missing something? We have a very simple site for a small business, not sure I can justify $99 per month for a service I don't really understand. Is this service more for SEO professionals than business owners? -Tom
Moz Pro | | TomHu0 -
Is it possible to do the comparision between a subfolder against a subdomain at SEOmoz campaign tool?
Is it possible to do the comparision between a subfolder against a subdomain at SEOmoz campaign tool? For instance: - I have a client who uses a subdomain model to render your homepage and he has a competitor hosted in a subfolder. It appears to be impossible to do that kind of comparision at the SEOmoz campaign tool. Do you have some other way to do that?
Moz Pro | | Abril0 -
SEOmoz vs Google Webmaster Tools on incoming links
I'm working on basic SEO for http://queueassoc.com. Google has indexed the non-www verions of the pages and these are what the SERPS return SEOmoz toolbar shows that all of the incoming links juice goes to the www. versions of the pages, none to the non-www version. Yesterday I set up GWMT for the site, submitted a sitemap with the www version of the pages and set the default address to the www version. I had to verify both versions of the site in order to do this and in looking at the non-www version I saw that Google had all the incoming links there and none in the www.version, the opposite of what SEOmoz shows. Is this just because Google only has the non-www versions in its index? Will they show the links to the www version once they get them in the index? I'm worried about losing Google Page Rank value or SEOmoz DA by making this switch.
Moz Pro | | bvalentine0 -
SEOMOZ Competitive Domain Analysis
Hi, I am just trying to get to the bottom of why I am seeing such wildly different figures in the SEOMOZ link analysis. The SEO company that we use is saying that we have around 331 external links to our website, when I check on Yahoo site explorer, it returns 264 external links, but in my SEOMOZ campaign manager, it says I have 4 external links? I fully appreciate that none of these methods are going to be wholly accurate, and are going to produce different results, but I can't understand why the SEOMOZ results are so vastly different. I also know for a fact there are a number of Followed links that I have added to a couple of websites, that Google has now indexed (as they appear in search results), but don't appear in the campaign manager results. If SEOMOZ use a different method to assess an external link, then no problem - my main concern here is that as a Newbie I am doing something wrong within the SEOMOZ tools, and don't want to take action based on results that may be incorrect because I have set things up incorrectly? Any help is greatly appreciated. Many thanks, Gareth
Moz Pro | | gdavies090319770 -
SEOmoz bar causes FF to hang
I use FireFox as my browser on a Windows pc. When I close FF it rarely closes properly. The process is still visible in the task manager. I need to end the process to shut it down. After researching the issue I learned this problem is usually caused by an add-on. I disabled my add-ons one at a time and it is clearly the SEOmoz bar causing the issue. I can run every other add-on without any problem but if I use the mozbar but itself, the issue occurs. I plan to report this problem to the help desk but first I wanted to ask if others are experiencing the same issue. The more data that can be collected, the easier it will be to resolve the problem. Thanks in advance for your feedback.
Moz Pro | | RyanKent0 -
SEOmoz bot and "noindex"
As a recent newbie to SEOmoz, I've been implementing some suggestions and doing a general tidy up. I removed URL's from our robots txt, and rolled out instead the noindex meta tag to pages we don't want indexed. But surprised to see issues that are now flagged from the last crawl by the moz bot from pages that have this meta tag? Does the SEOmoz bot not ignore this tag? Just want to make sure I've implemented it correctly, so the google bot does ignore it. Meta tag syntax is and is placed below the title tag. cheers Steve
Moz Pro | | sjr4x40 -
Why is Roger crawling pages that are disallowed in my robots.txt file?
I have specified the following in my robots.txt file: Disallow: /catalog/product_compare/ Yet Roger is crawling these pages = 1,357 errors. Is this a bug or am I missing something in my robots.txt file? Here's one of the URLs that Roger pulled: <colgroup><col width="312"></colgroup>
Moz Pro | | MeltButterySpread
| example.com/catalog/product_compare/add/product/19241/uenc/aHR0cDovL2ZyZXNocHJvZHVjZWNsb3RoZXMuY29tL3RvcHMvYWxsLXRvcHM_cD02/ Please let me know if my problem is in robots.txt or if Roger spaced this one. Thanks! |0