Crawl test from tools
-
Hi,
I notice that the crawl test which is from the Research Tools doesn't really get a new crawl even though there is 2 crawl per day. It will only provide the data which was acquire from the crawl diagnostics in my pro account. There is no point for me to get the data which I get from my crawl diagnostic isn't it? Even seomoz provided with more than 2 crawl per day also useless in this case.
This whole thing doesn't make sense as the crawl diagnostics will only perform a full crawl test once every week. but even the crawl test also not helping any thing out for me.
-
Everyone, thanks for helping... Greatly appreciated.
-
Hi there Hanzo,
Thanks for reaching out! it looks like you are having some trouble with the crawl test. Keri is correct that our crawl test uses "Cached" information to construct your crawl, which can take up to 2 days to clear. While I am not sure why the sites are still reporting 404s, there's a few things that could explain why you might be getting a stale crawl:
- Your server simply did not update the changes you've made on the backend.
- There might an issue with our crawler, which in my last status there isn't a outage that I know of.
I looked into your crawl data and the 404s seems to be confirmed by the report, if you download your latest report and do a sort you should see that
Hope I was able to provide some insight, please let me know if you have questions!
Best,
-
I believe the crawl test has a cache of a couple of days. I'm asking someone from the help desk to come in and confirm, but it may be Monday before they have a chance to do so.
-
I know there is third party app which I can use to crawl,
but then isn't the crawl test is there for us to do testing instead of using other app rather than seomoz's tool?
-
You may want to give screaming frog or xenu's link sleuth. They should be able to crawl you site and give you the info you need.
-
From what I do with the crawl test, it doesn't show me the latest crawl. I need to check whether all the error 404 or related error was being solved. It only show me the old data which was crawl before but not the latest one.
-
Hanzo, what is it that you need to do that the crawl test isn't helping you with?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unable to view crawl test
After doing a crawl test i get a download report. It then downloads in csv form and when I go to view it there is a curruption error or just a load of gibberish signs Can I not see the report onsite?
Moz Pro | | hantaah0 -
Have a Campaign, but only states 1 page has been crawled by SEOmoz bots. What needs to be done to have all the pages crawled?
We have a campaign running for a client in SEOmoz and only 1 page has been crawled per SEOmoz' data. There are many pages in the site and a new blog with more and more articles posted each month, yet Moz is not crawling anything, aside from maybe the Home page. The odd thing is, Moz is reporting more data on all the other inner pages though for errors, duplicate content, etc... What should we do so all the pages get crawled by Moz? I don't want to delete and start over as we followed all the steps properly when setting up. Thank you for any tips here.
Moz Pro | | WhiteboardCreations0 -
Crawl Errors from URL Parameter
Hello, I am having this issue within SEOmoz's Crawl Diagnosis report. There are a lot of crawl errors happening with pages associated with /login. I will see site.com/login?r=http://.... and have several duplicate content issues associated with those urls. Seeing this, I checked WMT to see if the Google crawler was showing this error as well. It wasn't. So what I ended doing was going to the robots.txt and disallowing rogerbot. It looks like this: User-agent: rogerbot Disallow:/login However, SEOmoz has crawled again and it still picking up on those URLs. Any ideas on how to fix? Thanks!
Moz Pro | | WrightIMC0 -
Adjusting SEOmoz Crawling Speed
How do you adjust the SEOmoz crawling speed? SEOmoz tried to crawl 10,000 pages in 3 hours and crashed our MySQL server.
Moz Pro | | cappuccino891 -
Keyword Difficulty Tool - How to use it the best way?
Hi, I am freshman, both, here on SEOmoz and in SEO generally and have a question concerning the assessment of KW difficulty. I did browse through the Q&A-Section (great content!!) but could not find a relevant answer to my problem. So I am currently building my initial keyword list, our website is only 2 months old, so we are still in the very early stage. Fashion is a very competitive area, so the KW Diff. Tool indicates high difficulty for a lot of words and phrases. however, I identified some with percentages <50 or even lower than that. Then I compared the results of the SEOmoz tool to Google Adwards and the Google results for competitiveness differed significantly. For example: for the KW Personal Shopping I got KWD from Seomoz 33% (in Germany) and from Google 0,5 for broad and 0,64 for exact search. I am quite confused how to make the right choices for my KWs now. Which metrics should I consider most? What else apart from the competition factor is behind the metric KW diff.? Does it matter in any way that I search from Germany for Germany in German? Do you have any further recommendations for the process of identifying the best Kws? Thanks a lot in advance, best from Berlin Tani
Moz Pro | | TaniBogi0 -
Seomoz research tools question for inbound links
What is the best seomoz tool and or indicator within a tool for checking on a link (directory or article site) to evaluate whether or not you would actaully want a link from them? Any way to see if google penalizes them for there tactics or if that can hurt me by them linking to me or whatever? I am at the beginning of a link building mission, but want to ensure that I do it methodically and correctly as possible based on the combines wisdom of this community. Thanks for your help, Steven
Moz Pro | | sfmatthews0 -
Which Google API do you use to creat a rank checking tool?
I can't seem to work this one out. Basically I want to make a very simple rank checker tool something similar to http://www.mikes-marketing-tools.com/ranking-reports/ where a user simply type in a domain and a keyword and in return is shown the position within the results. The problem is I can’t work out which API's I need in order to build it, does anyone know what I need to poke about with? It must work with Google and ideally with Bing and the rest
Moz Pro | | therealmarkhall0