Does SeoMoz realize about duplicated url blocked in robot.txt?
-
Hi there:
Just a newby question...
I found some duplicated url in the "SEOmoz Crawl diagnostic reports" that should not be there.
They are intended to be blocked by the web robot.txt file.
Here is an example url (joomla + virtuemart structure):
http://www.domain.com/component/users/?view=registration
and the here is the blocking content in the robots.txt file
User-agent: *
_ Disallow: /components/_
Question is:
Will this kind of duplicated url errors be removed from the error list automatically in the future?
Should I remember what errors should not really be in the error list?
What is the best way to handle this kind of errors?
Thanks and best regards
Franky
-
Hello Franky,
Yes, our crawler obeys robots.txt files. If you recently made that change to your robots then it should reflect in your next crawl. If this error doesn't go away, feel free to let us know help@seomoz.org. Thanks for letting us know!
-Abe
-
Don't be too worried about SEOMOZ's errors. Just be aware of them, and if you have done what you need to for the robots file in regards to S.E robots, they should take notice and there shouldn't be any issues. Always be sure to check GWT for errors, those are the ones you should fix asap.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicated content generated by keywords
Hello! I am kind of new to SEO and MOZ, so I really need your help to understand why some of my keywords generate duplicated content. Meaning, in my blog posts I use various SEO keywords. It shows up that in my MOZ crawl analysis, I have these keywords listed as duplicates: so two/three different keywords are pointing to the same articles and are considered duplicates? I really don't understand how it is possible. Did it also happen to you? I highly appreciate it. Thank you
Moz Pro | | DianaC0 -
Not provided Data in SEOMoz reports
Hi How do SeoMoz reports deal with 'not provided' data I see my total visits from organic search for a month are same as total of both my branded and non branded keyword traffic combined yet GA is reporting 157 visits from non provided data so is SEOMoz being very clever and finding a way to decipher this not provided data and allocate it accordingly in the reports ? Or if not what ? Many Thanks Dan
Moz Pro | | Dan-Lawrence0 -
Setting a campaign in SEOmoz
Hi All, when i set a new campaign i am asked to decide if to use: domin-name.com Or www.domain-name.com Can someone please explain the different in terms of the campaign If i use: domain-name.com - will the campaign run on www.domain-name.com too? Thank you SEOwise
Moz Pro | | iivgi0 -
Does SEOmoz have a Keyword Research tool similar to, say, the Google AdWords tool or the WebCEO Keyword Research Tool? And where might that be? (Sorry, I'm very new to SEOmoz Pro.)
I'm looking for an SEOmoz version of the classic WebCEO Keyword Research that would give you effective suggestions based on a keyword inquiry. I've made the switch from WebCEO, but I'm trying to find something similar to that Keyword Research tool. Am I going to just need to use the Google AdWords tool for this function or does SEOmoz have it's own version?
Moz Pro | | SmokewagonKen0 -
Does SEOMoz ever work?
Hi, I've signed up for the free 30 day trial and I'm on the edge of not actually subscribing to the service. I go through the Q & A boards which I find really interesting and hope I can add value there in the future, but the tools interest me more (and this is where the issue lies. Do they ever work? The Keyword difficulty tool just constantly says to come back in 20 minutes and I don't think the Rank Tracker has worked for at least half my freebie 30 days. Have the tools always been this flaky or is it a blip?
Moz Pro | | orlandovisiting1 -
SEOMOZ Crawl Test
Guys I really have an issue that i know have but cannot see if that makes sense. Basically 3 months ago i did a site wide 301 from economyleasinguk.co.uk to www.economy-car-leasing.co.uk Every thing looks good get all the correct header responses , all canonicals work perfectly , Google webmaster tools is updated fetch as google bot shows the old site is 301 I tried the seomoz crawl test today on the old domain and got this message Oh no! Looks like the page you were trying to access is temporarily down which at first thought ok because the site was not there it wont do it on an old 301 domain, however i tried it on a domain i know has just been 301'd and i got this message The URL http://www.site1.com/ redirects to http://site2.com/. Do you want to crawl http://site2.com/ instead?
Moz Pro | | kellymandingo
Would you like to:
Continue with www.site1.com
Continue with site2.com I really do not know what to do, its either the redirect script is missing something however its doing what it should or the server is a problem but again its doing what it should so why would SEOMOZ not be able to crawl the old URL like it example site above. Now the strange thing is Open Site Explorer does see the 301 and asks if i want to check the new URL instead Ps the redirect is done using PHP redirect which i am asking him to change to a htaccess as its now on a apache server and was wondering if this could be an issue, all pages go to correct pages as requested Thanks in Advance1 -
SEOMoz Keyword Limit
I am trying to add keywords to my SEOMoz campaign and it says I have reached my 300 KW limit eventhough I have only added 1 word! Any suggestions? uwt2V
Moz Pro | | theLotter0