Re:Storm's first birthday
by newobj on Wednesday September 19, 2012 @09:54PM
(#4546584)
What I'd like to understand at a glance about Storm is durability, idempotency, and specifically those two things in the face of all your standard failure modes.
Is Storm something that is going to get 99.9% of my data there which is good enough for some side-channel processing like ad targeting, or is it something that's D durable in the strongest sense of the word? If I'm ultimately delivering data to something that is 5 9's durable, or better yet S3 10-11 9's durable, is Storm going to be my "low point" of durability?
Re:Storm's first birthday
by djb_hackernews on Wednesday September 19, 2012 @10:18PM
(#4546646)
Read the wiki:
"Guarantees no data loss: A realtime system must have strong guarantees about data being successfully processed. A system that drops data has a very limited set of use cases. Storm guarantees that every message will be processed, and this is in direct contrast with other systems like S4."
Re:Storm's first birthday
by sanswork on Wednesday September 19, 2012 @03:19PM
(#4545074)
Nice to see this. I started trying to implement storm towards the beginning of this year since it seems like something I've wanted/reimplemented partially so many times and just found it to be so time consuming trying to get good resources or find a community for misc support that I gave up and reimplemented simple parts of it as I needed to.
I'm off the buy the books and bookmark this page for future reference. Great work Nathan.
Re:Storm's first birthday
by nathanmarz on Wednesday September 19, 2012 @04:26PM
(#4545413)
Sorry to hear that you had trouble finding support for your issues. The mailing list is very well trafficked so hopefully you have more luck there next time.
Re:Storm's first birthday
by sanswork on Wednesday September 19, 2012 @04:30PM
(#4545425)
No need to apologize, it was quite early in the development of the project(and expected). I'm just glad to see the progress you've made in a year and look forward to implementing Storm to solve a few of our issues in the near future.
Re:Storm's first birthday
by djb_hackernews on Wednesday September 19, 2012 @07:55PM
(#4546217)
Even if you have no use for storm, I'd strongly suggest you check out the wiki on github and browse the source code. It is IMO the nicest, cleanest, most exciting OSS project out there.
Re:Storm's first birthday
by djb_hackernews on Wednesday September 19, 2012 @10:15PM
(#4546635)
Oh easy questions:
The code itself is nice and clean, take a look yourself (chosen randomly) [1][2]
It's exciting because if you've ever done any real time processing of data streams at scale or even been daunted by the idea of it, and you read the wiki, it's clear storm is an exciting option. It does for Big Data Streams what Hadoop did for Big Data.
Kafka is interesting, but they don't really address similar use cases. They are more complimentary and in fact storm has a spout implementation for kafka[3].
Re:Storm's first birthday
by dkersten on Thursday September 20, 2012 @05:10AM
(#4547723)
They are more complimentary and in fact storm has a spout implementation for kafka
I would add to this that a lot of Storm users (myself included) use Storm together with Kafka - that is, Kafka is used to get data into (and possibly out of) Storm while Storm does the actual processing. Kafka is more along the lines of Kestrel and RabbitMQ.
Re:Storm's first birthday
by someone13 on Wednesday September 19, 2012 @11:25PM
(#4546783)
One of my major wishlist items for Storm is an end-to-end tutorial on how to get it working for a non-JVM language. I'm not a JVM coder, nor do I know much about it, and the documentation on how to get it working for Python is hard to follow at best, IMHO.
Re:Storm's first birthday
by encoderer on Wednesday September 19, 2012 @03:18PM
(#4545063)
Keep up the good work, Nathan. I presented Storm as a tech-talk at Formspring and no question, you made my job easy by building a platform that makes it easy to get a topology running without a ton of boilerplate.
Re:Storm's first birthday
by floppydisk on Wednesday September 19, 2012 @05:44PM
(#4545727)
I've used Storm for a couple of projects and really liked it. It significantly simplifies building real time data pipelines and makes deploying them KISS easy. I'm really excited to see some improved metrics coming out as well--if I had one gripe with Storm .7 it was that I wanted some more metrics to monitor cluster performance. Congrats on a year and keep up the good work, Storm is awesome!
Re:Storm's first birthday
by dkersten on Wednesday September 19, 2012 @06:04PM
(#4545799)
I've been using Storm for a few weeks and it works great, especially Trident, which I've been using for the past 2 or so weeks. Nathan has also been very helpful answering my n00b questions :)
Re:Storm's first birthday
by mattmcknight on Wednesday September 19, 2012 @11:41PM
(#4546829)
It is a little bit out of date, in that it doesn't cover anything in 8.0, but most of the basic stuff didn't change, it was more of new features being added.
Is Storm something that is going to get 99.9% of my data there which is good enough for some side-channel processing like ad targeting, or is it something that's D durable in the strongest sense of the word? If I'm ultimately delivering data to something that is 5 9's durable, or better yet S3 10-11 9's durable, is Storm going to be my "low point" of durability?
"Guarantees no data loss: A realtime system must have strong guarantees about data being successfully processed. A system that drops data has a very limited set of use cases. Storm guarantees that every message will be processed, and this is in direct contrast with other systems like S4."
https://github.com/nathanmarz/storm/wiki/Rationale
https://github.com/nathanmarz/storm/wiki/Trident-tutorial
Here's an explanation on how Trident achieves fully fault-tolerant, idempotent semantics:
https://github.com/nathanmarz/storm/wiki/Trident-state
I'm off the buy the books and bookmark this page for future reference. Great work Nathan.
https://github.com/nathanmarz/storm
Someone else posted link to Apache Kafka developed at LinkedIn.
http://incubator.apache.org/kafka/design.html
It seems like they address similar use cases... it would be interesting see a comparison.
The code itself is nice and clean, take a look yourself (chosen randomly) [1][2]
It's exciting because if you've ever done any real time processing of data streams at scale or even been daunted by the idea of it, and you read the wiki, it's clear storm is an exciting option. It does for Big Data Streams what Hadoop did for Big Data.
Kafka is interesting, but they don't really address similar use cases. They are more complimentary and in fact storm has a spout implementation for kafka[3].
[1] https://github.com/nathanmarz/storm/blob/master/src/clj/back... [2] https://github.com/nathanmarz/storm/blob/master/src/jvm/back... [3] https://github.com/nathanmarz/storm-contrib/tree/master/stor...
I would add to this that a lot of Storm users (myself included) use Storm together with Kafka - that is, Kafka is used to get data into (and possibly out of) Storm while Storm does the actual processing. Kafka is more along the lines of Kestrel and RabbitMQ.
Hmmm...
I am very interested in Storm, but wondering if the book is already out of date.