News for hackers. Stuff that matters.
Re:Infrastructure for Data Streams by felipesabino on Tuesday November 11, 2014 @01:39AM (#8588602)
My lazy self always wonder how nice it would be if some of these infrastructure designs were always accompanied with a docker/fig configuration example to be used as a start point/proof of concept for people looking for similar solutions.

It obviously happens some times [1] [2], but it should be more common...

[1] http://alvinhenrick.com/2014/08/18/apache-storm-and-kafka-cl...

[2] https://registry.hub.docker.com/u/ches/kafka/

Re:Infrastructure for Data Streams by erichmond on Tuesday November 11, 2014 @09:30AM (#8589931)
As you pointed out, this is already pretty common. As someone who's tried this before, it's hard to find the right balance. To setup a dead simple kafka config is about 10 lines of shell, so simple, it almost doesn't make sense to do.

A complex kafka setup is pretty involved (relatively speaking) and becomes domain specific pretty quickly, at which point, it probably becomes less usable / understandable to someone who is trying to learn / get interested.

On the whole, I agree with you. Some of the open source software that exists now is truly amazing and I think lots of people are defaulting to less-the-optimal solutions because they just don't have exposure to the latest and greatest.

Re:Infrastructure for Data Streams by ewencp on Tuesday November 11, 2014 @02:26AM (#8588724)
Agreed! It's not Docker, but I'm working on getting a decent Vagrant setup included with Kafka: https://issues.apache.org/jira/browse/KAFKA-1173 That supports pulling up a full cluster locally in VirtualBox or in EC2. Just a first cut, but it already makes testing a lot easier for me.

But in it's current state, that patch is a starting point that is really intended more for Kafka developers than for Kafka users. I really like what the Mesosphere folks have done -- great variety of OSes and cloud platforms, plus they do all the heavy lifting of bringing the cluster up for you.