Mac-Friendly, DOM-Rendering Spidering Tool for Multiple Users
-
Hello!
I am looking for a spidering tool that:
- Is Mac-friendly
- Can render the DOM and find JS links
- Can spider password-protected sites (prompts for password and then continues spider, etc.)
- Has competitive pricing for 8+ users.
Screaming Frog is amazing - and maybe we're just going to have to bite the bullet there. But if anyone has any other ideas, I've love to hear them. Thanks!
-
So - after digging around a lot and reading and re-reading every article that popped up for "screaming frog alternative", I've come to the conclusion that for the price, there really is nothing better than Screaming Frog right now.
I was impressed, however, with the incredibly helpful team from Deep Crawl. This enterprise tool is designed for larger websites - whereas Screaming Frog can crap out of your local machine runs out of memory. Because it's a more powerful tool, it's more expensive than Screaming Frog - but if you need an enterprise solution, it's definitely worth looking into. Another big differentiator is that Deep Crawl has no limit to the number of users, which is our primary pain point with Screaming Frog.
-
Right now we're updating SEOSpyder ( http://www.mobiliodevelopment.com/seospyder/ ) for rendering pages but i can't give you timeframe when will be done.
So far memory requirements isn't too high and was crawl 250k site with 8G ram machine.
-
Oh actually something I just realized is that potentially ScreamingFrog can do what you want and it will provide you with access to 8 users, but the setup is complicated. You would need to run it in a big virtual machine on AWS or Google Cloud Platform. That way you can scale the machine so it won't time out and everybody will still have access to it.
Back to your question: I've worked with Deepcrawl, a bit with Ryte and more with Botify. They're all great tools that are able to crawl your site. But you probably already looked into some of them.
-
Oh, interesting - can you help me understand about more about the cloud solution are you using...? Thanks!
-
Going to follow this, as I've been looking for something too. But we went the cloud service, as there is nothing that I acme across that can otherwise fulfill all these needs.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
One Article, Multiple(Same Keyword) Anchors, Same Urls
Hey Folks, So I have a 1000 word articles talking about say Dubai Holiday. Is it okay to have 4-5 Dubai Holiday as anchor linked to the same page. Or it should be only be used once.
Intermediate & Advanced SEO | | SAISEO0 -
Tool to identify if meta description are showing?
Hi we have a Ecommerce client with 1000s of meta descriptions, we have noticed that some meta descriptions are not showing properly, we want to pull and see which ones are showing on Google SERP results. You can use tools like screaming frog to pull meta description from page, but we want to see if it's showing for certain keywords. Any ideas on how to automate this? Cheers.
Intermediate & Advanced SEO | | brianna00 -
SEO Friendly Facets
Hi I'm still stuck on the subject if SEO friendly facets. Firstly, is it worth investing time in over things like SEO campaigns/content marketing as I'm the only one working on SEO and trying to prioritise all tasks 🙂 Can I set up facets so they are SEO friendly - should they simply be blocked? my concern is wasting crawl budget and duplicate pages. Here's an example of a page on the site - https://www.key.co.uk/en/key/lift-tables Here's an example of a facet URL - https://www.key.co.uk/en/key/lift-tables#facet:-1002779711011711697110,-700000000000001001651484832107103,-700000000000001057452564832109109&productBeginIndex:0&orderBy:5&pageView:list& What would be the best course of action to take to make them SEO friendly? Tips would be appreciated 🙂
Intermediate & Advanced SEO | | BeckyKey0 -
URL Parameter Setting Recommendation - Webmaster Tools, Breadcrumbs & 404s
Hi All, We use a parameter called "breadCrumb" to drive the breadcrumbs on our ecommerce product pages that are categorized in multiple places. For example, our "Blue Widget" product may have the following URLs: http://www.oursite.com/item3332/blue-widget
Intermediate & Advanced SEO | | Doug_G
http://www.oursite.com/item3332/blue-widget_?breadCrumb=BrandTree_
http://www.oursite.com/item3332/blue-widget_?breadCrumb=CategoryTree1_
http://www.oursite.com/item3332/blue-widget_?breadCrumb=CategoryTree2_ We use a canonical tag pointing back to the base product URL. The parameter only changes the breadcrumbs. Which of the following, if any, settings would you recommend for such a parameter in GWT: Does this parameter change page content seen by the user? Options: Yes/No
How does this parameter affect page content? Options: Narrows/Specifies/Other Currently, google decided to automatically assign the parameter as "Yes/Other/Let Googlebot Decide" without notifying us. We noticed a drop in rankings around the suspected time of the assignment. Lastly, we have a consistent flow of products that are discontinued that we 404. As a result of the breadcrumb parameter, our 404s increase significantly (one for each path). Would 800 404 crawl errors out of 18k products cause a penalty on a young site? We got an "Increase in '404' pages' email from GWT, shortly after our rankings seemed to drop. Thank you for any advice or suggestions! Doug0 -
New Google AdWords Keyword Tool - What Can We Do?
What options do we have for keyword research now that Google is switching from the Google AdWords Keyword Tool to the Keyword Planner??
Intermediate & Advanced SEO | | alhallinan0 -
Keyword tool for news?
Working on developing a news product and wondering if there are tools available to gauge search interest in a particular topic. For those that work in news, what are your favorite SEO tools?
Intermediate & Advanced SEO | | nicole.healthline0 -
How to solve a PHP problem to increase user experience ?
Hi all, I am having a peculiar PHP problem that is affecting the user experience of the site and thus, the SEO. The site is www.atlantiksurf.com and as you can see I use the QTranslate Plugin to manage the 3 different languages: www.atlantiksurf.com (Spanish) www.atlantiksurf.com/en www.atlantiksurf.com/de If you enter the german version: http://www.atlantiksurf.com/de/ , and you scroll down to the bottom of the page, you will notice there are a couple of posts that have (Español) in their Titles. This means that this particular post is written in Spanish and should only appear in the Spanish version of the site. Every single post is published on the three versions even when they are language orientated. The result is that when you press that specific Spanish title on the German version, you get nothing but this: http://www.atlantiksurf.com/de/aritz-aranburu-tendra-que-estar-un-mes-sin-competir/ Because obviously there is nothing there. If I go to the wordpress admin panel and search for the post manually, I can solve the problem by erasing all the default code of the different language versions that should not appear. But this is only a manual and non practical solution. The problem, I think, might be in the PHP orders that the Wordpress Theme is receiving from somewhere. I come to all of you SEomoz users as my last chance, because I've been months in discussions with Qtranslate users and php amateurs that cannot solve the issue. I know that this question isn't strictly about SEO, but in a way it is, because it must be affecting the way that Google look at us. Please, some help or orientation would be highly appreciated. rTqZI.png
Intermediate & Advanced SEO | | Tintanus0 -
Tool to check XML sitemap
Hello, Can anyone help me finding a tool to have closer look of the XML sitemap? Tks in advance! PP
Intermediate & Advanced SEO | | PedroM0