Re:Show HN: Interactive map for architecting big data pipelines
by lobster_johnson on Tuesday June 27, 2017 @12:50AM
(#14642657)
Thanks, that's helpful. Is building a pipeline with Java, consisting entirely of shell spouts, a viable option? Are there downsides to not using the Java API?
Also note that unlike Spark, Storm is a pure open source project that does not have a major commercial entity marketing its use cases. Hortonworks has put a little marketing effort behind it, but otherwise, it's just a mature & active Apache infrastructure project. Storm 2.0 is coming out soon and features a slew of performance- and reliability-improving enhancements.
But as for marketing buzz, Google has commercial reasons for you to use Beam and Dataflow, for example. And likewise Databricks for Spark.
It's probably a good idea to pick production large-scale data infrastructure on a metric other than recency of marketing buzz.
-$0.02 from one of the original authors of streamparse, the Python API for Storm