Ask HN: Fast, In-Memory, Distributed data analysis and machine learning?

Re:Ask HN: Fast, In-Memory, Distributed data analysis and ma... by agibsonccc on Saturday July 06, 2013 @11:56PM (#6000984)

I can vouch for storm. If only for the fact it's pretty easy to setup (especially compared to hadoop) Being able to leverage zookeeper for coordination allows you some extra capabilities for coordination as well. With that being said, just watch how you build your bolts/spouts. There's lots of ways you can send data in to the system, but in general , storm's documentation has been superb to work with.

I built a mini library for myself to auto construct the topologies based on a set of named dependencies to handle bolt/spout wiring. Aside from that, the builder interface for it is really nice if your data pipeline doesn't change.

There's good support for testing with a local cluster as well.

[ Parent | on HN ]

Re:Ask HN: Fast, In-Memory, Distributed data analysis and ma... by henrythe9th on Saturday July 06, 2013 @11:59PM (#6000989)

Thanks for your suggestion. Do you have any specific readings for me to look into for building bolts/spouts for sending data into the system?

Thanks

[ Parent | on HN ]

Re:Ask HN: Fast, In-Memory, Distributed data analysis and ma... by agibsonccc on Sunday July 07, 2013 @12:13AM (#6001021)

Here's the root wiki: https://github.com/nathanmarz/storm/wiki

Here's the system architecture: https://github.com/nathanmarz/storm/wiki/Concepts

Here's non JVM languages (specifically python) for building spouts/bolts https://github.com/nathanmarz/storm/wiki/Using-non-JVM-langu...

Here's an example project: https://github.com/nathanmarz/storm-starter

[ Parent | on HN ]

Re:Ask HN: Fast, In-Memory, Distributed data analysis and ma... by henrythe9th on Sunday July 07, 2013 @12:38AM (#6001060)

Thanks!

[ Parent | on HN ]