Mac-Friendly, DOM-Rendering Spidering Tool for Multiple Users
-
Hello!
I am looking for a spidering tool that:
- Is Mac-friendly
- Can render the DOM and find JS links
- Can spider password-protected sites (prompts for password and then continues spider, etc.)
- Has competitive pricing for 8+ users.
Screaming Frog is amazing - and maybe we're just going to have to bite the bullet there. But if anyone has any other ideas, I've love to hear them. Thanks!
-
So - after digging around a lot and reading and re-reading every article that popped up for "screaming frog alternative", I've come to the conclusion that for the price, there really is nothing better than Screaming Frog right now.
I was impressed, however, with the incredibly helpful team from Deep Crawl. This enterprise tool is designed for larger websites - whereas Screaming Frog can crap out of your local machine runs out of memory. Because it's a more powerful tool, it's more expensive than Screaming Frog - but if you need an enterprise solution, it's definitely worth looking into. Another big differentiator is that Deep Crawl has no limit to the number of users, which is our primary pain point with Screaming Frog.
-
Right now we're updating SEOSpyder ( http://www.mobiliodevelopment.com/seospyder/ ) for rendering pages but i can't give you timeframe when will be done.
So far memory requirements isn't too high and was crawl 250k site with 8G ram machine.
-
Oh actually something I just realized is that potentially ScreamingFrog can do what you want and it will provide you with access to 8 users, but the setup is complicated. You would need to run it in a big virtual machine on AWS or Google Cloud Platform. That way you can scale the machine so it won't time out and everybody will still have access to it.
Back to your question: I've worked with Deepcrawl, a bit with Ryte and more with Botify. They're all great tools that are able to crawl your site. But you probably already looked into some of them.
-
Oh, interesting - can you help me understand about more about the cloud solution are you using...? Thanks!
-
Going to follow this, as I've been looking for something too. But we went the cloud service, as there is nothing that I acme across that can otherwise fulfill all these needs.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does MOZ Have a Tool to Find a Competitors Broken Links?
I would like to use MOZ to identify a competitors broken links. Does MOZ have such a tool? Where is it? My SEO consultant suggests that we have bloggers write contend and then link it to our site. I am concerned about the quality of the links that will be generated. He is confident that they will be of high quality and the process will be relatively quick and easy. My concern is future spam penalty or low quality links. Another consultant suggested replacing a competitors broken links. How effective is this? Is it more labor intensive than other link building techniques? Thanks,
Intermediate & Advanced SEO | | Kingalan1
Alan Rosinsky1 -
Mega Good Bulk Keyword Tool
I have over 1000 keywords to sort out , I need a tool that picks up on variations and symmons. Does anyone know of a very good tool that is available? If not I'll do it the long way round with Excel 😞
Intermediate & Advanced SEO | | seoman100 -
Multiple Sitemaps Vs One Sitemap and Why 500 URLs?
I have a large website with rental listings in 14 markets, listings are added and taken off weekly if not daily. There are hundreds of listings in each market and all have their own landing page with a few pages associated. What is the best process here? I could run one sitemap and make each market's landing page .8 priority in the sitemap or make 14 sitemaps for each market and then have one sitemap for the general and static pages. From there, what would be the better way to structure? Should I keep all the big main landing pages in the general static sitemap or have them be at the top of the market segmented sitemaps? Also, I have over 5,000 urls, what is the best way to generate a sitemap over 500 urls? Is it necessary?
Intermediate & Advanced SEO | | Dom4410 -
Large Number of Links appearing in Google Webmaster Tools
Hello, In the last week we have noticed an extremely large number of backlink links appearing in Google Webmaster Tools. One of the sites which links to us now have over 101,000 backlinks pointing to us, when in reality it should only have 300-600. We have check the websites have not been hacked, with hidden links etc, but we can not find any. Has anyone else experienced problems with Google webmaster tools lately, displaying way too many links? Or could this be a negative SEO attack, which is yet to emerge. Thanks Rob
Intermediate & Advanced SEO | | tomfifteen0 -
Multiple Domain Names Pointing to One URL
Hi! A company I work with has purchased several (70-something) domain names that are relevant to their business. According to their IT pro, they're currently using DNS to point those domains to our IP address, with a catch-all header on IIS for that IP address. Essentially, we have 70-something domain names that direct to the homepage. I noticed that some have been indexed by Google and are pulling in the meta of the homepage they're being directed to. Is this potentially an issue? If so, would 301 redirects fix this or are we okay with the status quo and the indexing is no big deal? Thanks in advance!
Intermediate & Advanced SEO | | 199580 -
Experience with Google Disawow Tool and discovering bad back-links
Hi Community, is there any experience to tell here about the disawow tool from Google? Any review? It have helped revocer sites beaten by Penguin or penalized after WMT Unnatural Link building message? Which tools and methods you use to find bad back-links to submit for the disawow tool? Thanks for your feedback,
Intermediate & Advanced SEO | | Braumueller0 -
Multiple sites - ownership & link structure
Hi All I am in the process of creating a number of sites within the garden products sector; each site will have unique, original content and there will be no cross over. So for example I will have one on lawn mowers, one on greenhouses, another on garden furniture etc. My original thinking was to create a single limited company that would own each of the domains, therefore all the registrant details will be identical. Is this a sensible thing to do? (I want to be totally white hat) And what, if any, are the linking opportunities between each of the sites? (16 in total). Not to increase ranking, more from an authoritative perspective. And finally, how should I link between each site? Should I no follow the links? Should I use keyword contextual links? Any advice ideas would be appreciated 🙂 Please note: It has been suggested that I just create one BIG site. I've decided against this as I want to use the keyword for each website in the domain name as I believe this still has value. Thanks
Intermediate & Advanced SEO | | danielparry0 -
Multiple IPs (load balancing) for same domain
Hello, I'm considering moving our main website to a multiple servers, perhaps in multiple different datacenters and use a DNS round robin load balancing by assigning it 4 different IP addresses (probably from 4 different C classes). example:
Intermediate & Advanced SEO | | maddogx
ourdomain.com A 1.1.1.1
ourdomain.com A 2.2.2.2
ourdomain.com A 3.3.3.3
ourdomain.com A 4.4.4.4 Every time you ping the domain you will get a response from another IP of the group. Therefore search engines will see a different IP each time they scan the site. We have used the main IP for our website for past 6 years without changing it. We have a quite good SEO in our niche which I don't want to loose of course. My question is, will adding more IPs to the domain affect any how on the ranking ? What is the suggested way to do it anyway? What is recommended to do before and after? Thanks for you attention and help in advance. Dmitry S.0