Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2

Re:Using Spark and Zeppelin to Process Big Data on Kubernete... by mindcrime on Wednesday March 30, 2016 @04:45PM (#11392852)

I really wish there were more tutorials like this on how to set up Spark and other Big Data tools (TensorFlow) on cloud computing, as that personally has been the primary barrier to starting work with extra large amounts of data.

Just as an FYI, we[1] are working on an open source, cloud based Machine Learning / Big Data platform that might be of interest to you. It's not all ready yet, but when it is, there will be a simple REST API that allows you to define the kind of setup you want, "push a button" and have it all deployed. Our initial backend is AWS with plain jane EC2 nodes, but it will be possible to extend it to other configurations as well.

Right now we deploy a Spark/Hadoop Cluster with Apache SystemML, Mahout and MLLib installed. Zeppelin will be coming to the stack, as will other tools like TensorFlow, SparkR, CaffeOnSpark, etc.

We'll be offering our own hosted service based on this, but it'll be open source so you can deploy it in an environment of your own if you wish.

We'll do a "Show HN" when we have something ready, so keep an eye out if that sounds interesting.

I also plan to write up some tutorial and documentation based on our experiences building this out, but the priority right now is getting it built. :-)

[1]: http://www.fogbeam.com

[ Parent | on HN ]

Re:Using Spark and Zeppelin to Process Big Data on Kubernete... by TheIronYuppie on Wednesday March 30, 2016 @05:07PM (#11393012)

Disclosure: I work at Google on Kubernetes.

If you'd like to do this in containers/Kubernetes, we'd love to highlight your work! Kubernetes runs great on AWS (as well as GCP, Azure and elsewhere), so no cloud migration required.

[ Parent | on HN ]

Re:Using Spark and Zeppelin to Process Big Data on Kubernete... by jaz46 on Wednesday March 30, 2016 @06:21PM (#11393492)

Shameless plug: Pachyderm is another way to run big data workloads in Kubernetes. github.com/pachyderm/pachyderm. We spoke at KubeCon SF last year and v1.0 is coming out next month!

[ Parent | on HN ]

Re:Using Spark and Zeppelin to Process Big Data on Kubernete... by TheIronYuppie on Wednesday March 30, 2016 @06:43PM (#11393600)

Disclosure: I work at Google on Kubernetes

Great to hear, congrats on reaching 1.0! Please do reach out when you get there, we'd love to show off your work to the community.

[ Parent | on HN ]

Re:Using Spark and Zeppelin to Process Big Data on Kubernete... by mindcrime on Wednesday March 30, 2016 @05:25PM (#11393150)

Awesome, we'll definitely be looking into that as we progress. "Version 1" is going straight to raw VM's as we saw it as the path of least resistance to getting an MVP out (esp. since nobody here has done much work with containers yet) but the container route is definitely something we'll be pursuing at some point.

[ Parent | on HN ]