Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2

Re:Using Spark and Zeppelin to Process Big Data on Kubernete... by ryanl-ee on Wednesday March 30, 2016 @03:55PM (#11392417)

Have you checked out DCOS? Once you have your cluster set up, your path to installing Spark is running "dcos package install spark" on your laptop.

[ Parent | on HN ]

Re:Using Spark and Zeppelin to Process Big Data on Kubernete... by minimaxir on Wednesday March 30, 2016 @04:10PM (#11392556)

Huh, no I haven't. Although the package support appears to be more narrow than with Kubernetes/Docker, but DCOS is a possibility too.

[ Parent | on HN ]

Re:Using Spark and Zeppelin to Process Big Data on Kubernete... by ryanl-ee on Thursday March 31, 2016 @11:01AM (#11397657)

Well, the included packages in the "dcos package" CLI repo are just the prepackaged frameworks. You can actually deploy Kubernetes on top of DCOS in the same way - it natively supports Docker containers as an execution method.

[ Parent | on HN ]

Re:Using Spark and Zeppelin to Process Big Data on Kubernete... by ptrincr on Thursday March 31, 2016 @01:57AM (#11395263)

You should also check out Ambari, fantastic open source Hadoop/spark cluster management tool.

[ Parent | on HN ]

Re:Using Spark and Zeppelin to Process Big Data on Kubernete... by mindcrime on Thursday March 31, 2016 @04:24AM (#11395737)

Second the motion. Ambari is actually one of the key elements of the tech stack we're using for our service. With the "Blueprints" feature and the REST API, it's a really convenient way to automate the provisioning of Spark/Hadoop clusters.

Even better, Ambari isn't actually limited to installing just Hadoop/Spark, etc. In principle, you could extend it to take care of installing pretty much anything.

[ Parent | on HN ]