Does SeoMoz realize about duplicated url blocked in robot.txt?
-
Hi there:
Just a newby question...
I found some duplicated url in the "SEOmoz Crawl diagnostic reports" that should not be there.
They are intended to be blocked by the web robot.txt file.
Here is an example url (joomla + virtuemart structure):
http://www.domain.com/component/users/?view=registration
and the here is the blocking content in the robots.txt file
User-agent: *
_ Disallow: /components/_
Question is:
Will this kind of duplicated url errors be removed from the error list automatically in the future?
Should I remember what errors should not really be in the error list?
What is the best way to handle this kind of errors?
Thanks and best regards
Franky
-
Hello Franky,
Yes, our crawler obeys robots.txt files. If you recently made that change to your robots then it should reflect in your next crawl. If this error doesn't go away, feel free to let us know help@seomoz.org. Thanks for letting us know!
-Abe
-
Don't be too worried about SEOMOZ's errors. Just be aware of them, and if you have done what you need to for the robots file in regards to S.E robots, they should take notice and there shouldn't be any issues. Always be sure to check GWT for errors, those are the ones you should fix asap.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Tags on my website cause duplicate content
Hi I just recently started a website and I am new to MOZ pro. What Moz pro detected on my website under high priority is that "duplicate page content" and what I realize about these duplicate page content is regarding the tags i put on my post. Because it is a wordpress blog, we are allow to add tags on the side before we publish our post. And because of these tags, it linked to the same page but different url. for example website.com/tags/whatever website.com/tags/whatever 2 and both these url direct to the same page So how do i solve this? do i just stop tagging whenever i write a post? delete all tags while it is not necessary? i seen method like 301 redirect or rel=canonical but is there anyway to solve this problem so I do not face this issue whenever i make a new post in my blog? I mean it doesnt make sense to redirect 301 to every single tags i have whenever i write a new post right? thanks guys
Moz Pro | | andzon0 -
Getting my top keywords separated out in SEOmoz reports
I am using the standard functionality to produce weekly Moz reports. There does not seem to be a setting to show rankings of my most important keywords. It would be nice to have those high-level keywords on the first page. For example, I have 200 keywords in an account. I want the report to show on a page my 10 most important keywords. Is there a way to set up a Label to my keywords in order to product a report page just for those keywords?
Moz Pro | | clicktoshop0 -
What user agent is used by SEOMOZ crawler?
We have a pretty tight robots.txt file in place to only allow the major search engines. I do not want to block SEOMOZ.ORG from being able to crawl the site so I want to make sure the user agent is open.
Moz Pro | | eseider0 -
Seomoz keyword research
Hi, does seomoz have keyword research tool? or any plan to have one in near future?
Moz Pro | | imsameer0 -
Port 80 and Duplicate Content
The SEOmoz Web App is showing me that every single URL on one of my clients' domains has a duplicate in the form of the URL + :80. For instance, the app is showing me that www.example.com/default.aspx is duplicated in the form of www.example.com:80/default.aspx Any idea if this is an actual problem or just some kind of reporting error? Any help would be appreciated.
Moz Pro | | AnthonyMangia0 -
Redirecting duplicate .asp pages??
Hi all, I have a bit of a problem with duplicate content on our website. The CMS has been creating identical duplicate pages depending on which menu route a user takes to get to a product (i.e. via the side menu button or the top menu bar). Anyway, the web design company we use are sorting it out going forward, and creating 301 redirects on the duplicate pages. My question is, some of the duplicates take two different forms. E.g. for the home page: www.<my domain="">.co.uk
Moz Pro | | gdavies09031977
www..<my domain="">.co.uk/index.html
www.<my domain="">.co.uk/index.asp</my></my></my> Now I understand the 'index.html' page should be redirected, but does the 'index.asp' need to be directed also? What makes this more confusing is when I run the SEOMoz diagnostics report (which brought my attention to the duplicate content issue in the first place - thanks SEOMoz), not all the .asp pages are identified as duplicates. For example, the above 'index.asp' page is identified as a duplicate, but 'contact-us.asp' is not highlighted as a duplicate to 'contact-us.html'? I'm a bit new to all this (I'm not a IT specialist), so any clarification anyone can give would be appreciated. Thanks, Gareth0 -
Why is Roger crawling pages that are disallowed in my robots.txt file?
I have specified the following in my robots.txt file: Disallow: /catalog/product_compare/ Yet Roger is crawling these pages = 1,357 errors. Is this a bug or am I missing something in my robots.txt file? Here's one of the URLs that Roger pulled: <colgroup><col width="312"></colgroup>
Moz Pro | | MeltButterySpread
| example.com/catalog/product_compare/add/product/19241/uenc/aHR0cDovL2ZyZXNocHJvZHVjZWNsb3RoZXMuY29tL3RvcHMvYWxsLXRvcHM_cD02/ Please let me know if my problem is in robots.txt or if Roger spaced this one. Thanks! |0 -
SEOmoz mobile search keyword tracking?
Does anyone know if we can expect keyword rank tracking in Google mobile? It would be a cool feature for SEOmoz Pro. Does anybody know of any reliable software that does this at the moment?
Moz Pro | | A_Q0