Screaming Frog, Moz and other crawlers
-
Hi
Ignorant question, but is it possible to use Screaming Frog or the Moz crawler or any other reputable crawler for a site still in development i.e. it is yet to be indexed? If so, could someone provide some quick instructions on how this can be done.
Thanks in advance for any support.
Neil
-
Thanks for the answer, much appreciated.
-
Yes, thanks for the answer.
-
Tim,
As long as you can get to the site remotely via a website address you should be good to go. However, if the site is blocking crawlers via robots.txt file or meta robots tag rogerbot won't access it. On the other hand, screamingfrog has a setting to tell it to ignore the robots.txt file if one exists.
-
If the site is online or on a domain or somewhere where Screaming Frog can reach it you can spider it with the software yes.
But it has to be online somewhere so the software can reach it.
Did this answer your question?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Why is my website Backlinks Lower on Moz than on SEMRush?
Hi, I'm new to Moz and I am trying to find out why my website Birmingham Roofers has 7 backlinks on Moz it has 21 on SEMRush. However, I read somewhere online in one of your blog post that it crawls in real time.
Moz Bar | | ajegunle1 -
Why is the Moz tool bar page analysis saying my website is from Romania when we are in the United States?
So when I go to my client's website, https://www.paracore.com/ and on the home page, I use the Moz toolbar. From there I then use the "Page Analysis." If you look at the "URL" line there is a Romanian flag next to the site name. Then I scroll down within the page analysis and the "Country" line says Romania. This is a WordPress site, and the company is based in Arizona. Can anyone explain to me if this is code that I can find and change or remove? Any insight would be greatly appreciated.
Moz Bar | | Striventa1 -
Moz Crawler Causing Server Timeouts... Crawling thousands of non-existant pages with query parameters
Moz crawler is crawling all pages like this: http://www.xxxx.com/?product_count=100&product_order=desc&product_orderby=date http://www.xxxx.com/?product_count=100&product_order=desc&paged=1 http://www.xxx.com/?product_count=100&product_order=desc&product_view=grid Last month it crawled 80,000 pages on a site with less than 100 pages. Is there a way to select only certain pages to be crawled? Right now it is still crawling this site, since Monday morning and it's Tuesday mid-day. Every Monday it is causing time-outs from high band width on our server. Just getting ready to delete this client from the account unless there is a solution someone can give us. Thanks.
Moz Bar | | adirondack0 -
Moz Trust do not display
Hello, I just see there is some problem in seo moz even pro membership. I can not see Moz Trust to check trust ratio.How I can do it if it is blank. I also do not find a Single Question example for Example Trust Link profile I see there is a similar post by majestic but not in Seo moz. I would like to know similar details like majestic showing the example good site or bad site.Example reference below.
Moz Bar | | rarcntv
https://blog.majestic.com/general/using-majestic-citation-flow-trust-flow-check-quality-link-prospects/ So I see similar example post that is atleast help me to understand the depth of quality link profile and bad link profile point of view from seo moz. Thanks0 -
Why isn't the Moz bar data populating for Yahoo sites?
The Moz bar isn't populating information for Yahoo homepage or it's verticals (i.e. homes, autos, finance, etc.), but I can get this data for other portals like AOL or MSN. I'm specifically looking for PA, mR, and DA information, but instead I get a generic "Search Profile" bar with no page/site-specific data.
Moz Bar | | AllieBell
Is there a reason Open Site Explorer data isn't populating for this particular portal?0 -
How to make sense of data for this keyword (Moz Keyword vs. Google KW Planner)
The keyword "auto immune disorder" has an avg monthly (local/US) search of 880 in Google KW Planner with "low" competition In Moz, the same keyword has an avg monthly (local/US) search of 5 with a KW difficulty of 71%. I realize that the competition in Google KWP is based on ads, but is there a discrepancy when the search volume is so different in Google vs. Moz/Bing? Based on the above data, it seems like it would make sense to do a Google Adword (high search volume, low competition) instead of trying to rank organically...am I on the right track here? Thanks!
Moz Bar | | lulu710 -
Confusing Moz Crawl?
Hi there, I am not sure if I am missing on something but the moz crawls are rather confusing. After singing in I have received 11 emails with crawls and today I have received again new, When I go to check there to the dashboard it shows 26 pages with issues. When I scroll down I see the pages with issue. Then when I click on the first page listed, to view the issues it says this: Rel Canonical
Moz Bar | | Rebeca1
Using rel=canonical suggests to search engines which URL should be seen as canonical. For this site: http://villasdiani.com/ but we have sorted out the canonical issues a long time ago. Is this a wrong information or is it really true that we do not specify the canonical for our site? Then the second page with issue is there listed http://villasdiani.com/beach-villas/ and it says: Duplicate Page Title
You should use unique titles for your different pages to ensure that they describe each page uniquely and don't compete with each other for keyword relevance. But it does not point out which page is duplicate with this one! I do not have any other page named the same way. It also says in Issues overview 26pages with issues, but it shows on the bottom only 5 under and when I click on view more it brings me to high priority issues where is 0. The most is freaking me out this report: When I click on links, there are listed on the bottom the pages with highest authority among which I found this http://villasdiani.com/db I have never created this kind of page! Funny enough when I click on it it really open that page! How this can be??? In issues overview it also shows on the bottom, right corner 11 page with duplicate content but when I click on it to review it it brings me to high priority issues windows where is not displayed anything Can somebody advice me regarding of this. I have sign up here to learn and sort out the problems with the site but so far I am only getting more confused here. Thank you very much for looking into this.0 -
Rel Canonical and Moz Crawl
we have Rel Canonical tags set up on a few pages. When viewing the page source, the tags are correct. However, Moz Crawl results show the opposite. for example the page source, correctly shows, URL X with a Rel canonical Tag of URL Y
Moz Bar | | S.S.N
but.. Moz crawl is showing URL Y with a Rel Canonical Tag of URL X ..any thoughts why this would happen? which should i trust more?0