Mac-Friendly, DOM-Rendering Spidering Tool for Multiple Users
-
Hello!
I am looking for a spidering tool that:
- Is Mac-friendly
- Can render the DOM and find JS links
- Can spider password-protected sites (prompts for password and then continues spider, etc.)
- Has competitive pricing for 8+ users.
Screaming Frog is amazing - and maybe we're just going to have to bite the bullet there. But if anyone has any other ideas, I've love to hear them. Thanks!
-
So - after digging around a lot and reading and re-reading every article that popped up for "screaming frog alternative", I've come to the conclusion that for the price, there really is nothing better than Screaming Frog right now.
I was impressed, however, with the incredibly helpful team from Deep Crawl. This enterprise tool is designed for larger websites - whereas Screaming Frog can crap out of your local machine runs out of memory. Because it's a more powerful tool, it's more expensive than Screaming Frog - but if you need an enterprise solution, it's definitely worth looking into. Another big differentiator is that Deep Crawl has no limit to the number of users, which is our primary pain point with Screaming Frog.
-
Right now we're updating SEOSpyder ( http://www.mobiliodevelopment.com/seospyder/ ) for rendering pages but i can't give you timeframe when will be done.
So far memory requirements isn't too high and was crawl 250k site with 8G ram machine.
-
Oh actually something I just realized is that potentially ScreamingFrog can do what you want and it will provide you with access to 8 users, but the setup is complicated. You would need to run it in a big virtual machine on AWS or Google Cloud Platform. That way you can scale the machine so it won't time out and everybody will still have access to it.
Back to your question: I've worked with Deepcrawl, a bit with Ryte and more with Botify. They're all great tools that are able to crawl your site. But you probably already looked into some of them.
-
Oh, interesting - can you help me understand about more about the cloud solution are you using...? Thanks!
-
Going to follow this, as I've been looking for something too. But we went the cloud service, as there is nothing that I acme across that can otherwise fulfill all these needs.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Event Schema for Multiple Occurrences
I am wondering the best way to mark up an event page with multiple occurrences. For example, we have an event that happens over the course of 4 sequential weekends:
Intermediate & Advanced SEO | | Your_Workshop
9/28-9/29
10/5-10/6
10/12-10/13
10/19-10/20 Our website allows us to enter multiple occurrences that results in a single event listing page which outputs all dates (to eliminate duplicate content, titles, metas, etc.) but allows each occurrence to output individually on our events calendar in the respective individual date. Each time the event is shown, it links to the same listing page. I am wondering if we can add event schema on a single listing multiple times to cover each occurrence. In the above example, we would have 4 schemas on the listing page for each date range/weekend. In our current schema, we end up with a start and end date identified as 9/28-10/20 but it is not clear that the event is just happening on the weekends with gaps in between. Any suggestions are welcome however, we are really trying to NOT list each as an individual event on the website both for the duplicate content issue and the extra burden on our client that lists events for a very large geographic area.0 -
Tool for user intent
Hello, Is there a tool that can tell me what the user intent of my keyword is and how I should present my page (the type of content users want to see it, what questions they want answered ? Thank you,
Intermediate & Advanced SEO | | seoanalytics0 -
One domain - Multiple servers
Can I have the root domain pointing to one server and other URLs on the domain pointing to another server without redirecting, domain masking or HTML masking? Dealing with an old site that is a mess. I want to avoid migrating the old website to the new environment. I want to work on a page by page and section by section basis, and whatever gets ready to go live I will release on the new server while keeping all other pages untouched and live on the old server. What are your recommendations?
Intermediate & Advanced SEO | | Joseph-Green-SEO0 -
Is Paging Comments SEO Friendly? Implications?
So glad to be here. Just amazed to see so many discussions over here. I had a quick query. One of our my blog has more than 500 comments on almost 50+ posts (and in some posts, it's even 1000+ comments). This impacts the load time as well as the experience on mobile. So, wanted to understand if I enable pagination of comments, is it SEO friendly. Does it negatively impacts SEO? I do not want to take the route of migrating to Disqus, FB comments, etc.
Intermediate & Advanced SEO | | flmgo820 -
User generated content - manual warning from Google
Over the weekend our website received large amounts of spammy comments / user profiles on our forums. This has led to Google giving us a partial manual action until we clear things up. So far we have: Cleared up all the spam, banned the offending user accounts, and temporary enabled admin-approval for new sign ups. We are currently investigating upgrading the forum software to the latest version in order to make the forums less susceptible to this kind of attack. Could anyone let me know whether they think it is the right time for us to submit a reconsideration request to get the manual action removed? Will the temporary actions we have taken be enough to get the ban lifted, or should we wait until the forum software has been updated? I'd really appreciate any advice, especially if there is anyone here who has experienced this issue themselves 🙂
Intermediate & Advanced SEO | | RG_SEO0 -
301 Redirecting multiple domains to brand new domain
Hi guys, I have read quite a bit of stuff on 301 redirects after Penguin. Hoping someone could help me out. im looking at a way to do a legit 301 redirect without passing the penalty. I have acquired two businesses, business1 and business2, that both had websites that were hit by penguin. Ive anaylsed there backlinks and theres a lot of spammy forum links and comments and I was also informed they were both using buildmyrank. A side note, buiness2 only started using BMR after it noticed business1 have large amounts of high PR links. business1.com was ranking at position 1 till the penguin hit. Business2.com was ranking around page 2 I work in the same arena as these two businesses and didnt generate any business via the internet. When these 2 businesses failed (due to loss of rankings and traffic) i decided to take them over. What I am thinking of doing is 301'ing both business domains to my brand new, zero links, domain which will be the name of my new company. I will combine the content from both sites, around 1000 pages, in to the new one. So my question is, does 301'ing multiple domains, that target the same keywords, and operate in the same niche, look less "spammy" then 301'ing 1 domain? I'm trying to look at it in the eyes of google. It is a legit merging of businesses. Thanks for your help, really appreciate your time
Intermediate & Advanced SEO | | JohnPeters0 -
Multiple URL's exist for the same page, canonicaliazation issue?
All of the following URL's take me to the same page on my site: 1. www.mysite.com/category1/subcategory.aspx 2. www.mysite.com/subcategory.aspx 3. www.mysite.com/category1/category1/category1/subcategory.aspx All of those pages are canonicalized to #1, so is that okay? I was told the following my a company trying to make our sitemap: "the site's platform dynamically creates URLs that resolve as 200 and should be 404. This is a huge spider trap for any search engine and will make them wary of crawling the site." What would I need to do to fix this? Thanks!
Intermediate & Advanced SEO | | pbhatt0 -
Multiple Domain names pointing at one website
Hello, A collegue has asked if we can buy multiple domain names which contain keywords and point them at our website. Is this good practise or will it be seen as spam? Will these domains actually get ranked? I'm sure I'm not the first person to raise this but can't seem to find any questions and answers about this. Thanks Mark
Intermediate & Advanced SEO | | markc-1971830