Mac-Friendly, DOM-Rendering Spidering Tool for Multiple Users
-
Hello!
I am looking for a spidering tool that:
- Is Mac-friendly
- Can render the DOM and find JS links
- Can spider password-protected sites (prompts for password and then continues spider, etc.)
- Has competitive pricing for 8+ users.
Screaming Frog is amazing - and maybe we're just going to have to bite the bullet there. But if anyone has any other ideas, I've love to hear them. Thanks!
-
So - after digging around a lot and reading and re-reading every article that popped up for "screaming frog alternative", I've come to the conclusion that for the price, there really is nothing better than Screaming Frog right now.
I was impressed, however, with the incredibly helpful team from Deep Crawl. This enterprise tool is designed for larger websites - whereas Screaming Frog can crap out of your local machine runs out of memory. Because it's a more powerful tool, it's more expensive than Screaming Frog - but if you need an enterprise solution, it's definitely worth looking into. Another big differentiator is that Deep Crawl has no limit to the number of users, which is our primary pain point with Screaming Frog.
-
Right now we're updating SEOSpyder ( http://www.mobiliodevelopment.com/seospyder/ ) for rendering pages but i can't give you timeframe when will be done.
So far memory requirements isn't too high and was crawl 250k site with 8G ram machine.
-
Oh actually something I just realized is that potentially ScreamingFrog can do what you want and it will provide you with access to 8 users, but the setup is complicated. You would need to run it in a big virtual machine on AWS or Google Cloud Platform. That way you can scale the machine so it won't time out and everybody will still have access to it.
Back to your question: I've worked with Deepcrawl, a bit with Ryte and more with Botify. They're all great tools that are able to crawl your site. But you probably already looked into some of them.
-
Oh, interesting - can you help me understand about more about the cloud solution are you using...? Thanks!
-
Going to follow this, as I've been looking for something too. But we went the cloud service, as there is nothing that I acme across that can otherwise fulfill all these needs.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Matching user intent in my blog
Hello, I am planning on doing a blog on travel bike basic. I noticed that the google keyword tool gives me things like How to plan a bike route Can I bike during pregnancy etc... In order to compete on that keyword do I need to answer those questions or can I answer different ones and still rank such as : The tool kit that are recommend. Whether you should take an insurance or not, if so which one. Thank you,
Intermediate & Advanced SEO | | seoanalytics0 -
One Article, Multiple(Same Keyword) Anchors, Same Urls
Hey Folks, So I have a 1000 word articles talking about say Dubai Holiday. Is it okay to have 4-5 Dubai Holiday as anchor linked to the same page. Or it should be only be used once.
Intermediate & Advanced SEO | | SAISEO0 -
Should I care about this Webmaster Tools Message
Here is the message: "Googlebot found an extremely high number of URLs on your site: http://www.uncommongoods.com/" Should i try to do anything about this? We are not having any indexation issues so we think Google is still crawling our whole site. What could be some possible repercussions of ignoring this? Thanks Mozzers! -Zack
Intermediate & Advanced SEO | | znotes0 -
301 Redirecting Multiple Domains
I have several complete websites with blogs setup for different geo locations and was considering forwarding them all to one domain directly would greatly benefit ranking. The blogs are all linked together and that is where most of the links come from. Would I benefit in 301 Redirecting the domains?
Intermediate & Advanced SEO | | WindshieldGuy-2762210 -
Tool that can retrieve mysite URL's
Hi, Tool that can retrieve mysite URL's I am not talking about href,open explorer, Majestic etc I have a list of 1000 site URL's where my site name is mentioned. I want to get the exact URL of my site next to the URL i want to query with Example http://moz.com/community is the URL i have and if this page has mysite name then i need to get the complete URL captured. Any software or tool that can do this? I used one for sure which got me this info but now i don't remember it Thanks
Intermediate & Advanced SEO | | mtthompsons0 -
SEO Moz Keyword Ranking Tool
The SEO Moz keyword ranking tool is useful and fairly accurate but it would be more useful to know why the ranking changed. Can the tool provide any insight in this regard?
Intermediate & Advanced SEO | | casper4340 -
Tool to check XML sitemap
Hello, Can anyone help me finding a tool to have closer look of the XML sitemap? Tks in advance! PP
Intermediate & Advanced SEO | | PedroM0 -
Migrating multiple sites and trying to save link juice
I have an interesting problem SEOmozers and wanted to see if I could get some good ideas as to what I should to for the greatest benefit. I have an ecommerce website that sells tire sensors. We just converted the old site to a new platform and payment processor, so the site has changed completely from the original, just offering virtually the same products as before. You can find it at www.tire-sensors.com We're ranked #1 for the keyword "tire sensors" in Google. We sell sensors for ford, honda, toyota, etc -- and tire-sensors.com has all of those listed. Before I came along, the company I'm working for also had individual "mini ecommerce" sites created with only 1 brand of sensors and the URL to match that maker. Example : www.fordtiresensors.com is our site, only sells the Ford parts from our main site, and ranks #1 in Google for "ford tire sensors" I don't have analytics on these old sites but Google Keyword Tool is saying "ford tire sensors" gets 880 local searches a month, and other brand-specific tire sensors are receiving traffic as well. We have many other sites that are doing the same thing. www.suzukitiresensors.com (ranked #2 for "suzuki tire sensors") Only sells our Suzuki collection from the main site's inventory etc We need to get rid of the old sites because we want to shut down the payment gateway and various other things those sites are using, and move to one consolidated system (aka www.tire-sensors.com) Would simply making each maker-specific URL (ie. fordtiresensors.com) 301 redirect to our main site (www.tire-sensors.com) give us to most benefit, rankings, traffic etc? Or would that be detrimental to what we're trying to do -- capturing the tire sensors market for all car manufacturers? Suggestions? Thanks a lot in advance! Jordan
Intermediate & Advanced SEO | | JordanGodbey0