News for hackers. Stuff that matters.
Re:Generating Recommendations at Amazon Scale with Apache Sp... by minimaxir on Sunday July 10, 2016 @01:33PM (#12066339)
Setting up Spark clusters is easy relative to setting up clusters, but not easy enough yet to set up and configure relative to simply downloading a package in R/Python.
Re:Generating Recommendations at Amazon Scale with Apache Sp... by nchammas on Sunday July 10, 2016 @01:55PM (#12066439)
Well, the absolute easiest way to run Spark is to do it locally (e.g. you can brew install it on a Mac and just go) or to pay for a proprietary service like Databricks, which makes setting up a cluster take a few clicks.

That said, I think `flintrock launch my-cluster` is almost as easy as doing `pip install ...`.

You do need an AWS account and you do need to set your preferences like region and key name in a config file, but I don't see how you can get out of doing even that without subscribing to some managed service like Databricks that abstracts everything away and replaces it with a nice Web UI.

Re:Generating Recommendations at Amazon Scale with Apache Sp... by mgummelt on Monday July 11, 2016 @12:15PM (#12072080)
If you're running a DC/OS cluster, it actually is!

> dcos package install spark

This gives you a Spark CLI, and installs the Spark Cluster Dispatcher so you can run jobs async. It also has options for installing the history server, and for SSL/Kerberos support.

Disclaimer: I work at Mesosphere and contribute to the DC/OS Spark package.