If you're on AWS, it's already quite easy to set up a cluster today, no?
There's EMR of course, and there are tools like spark-ec2 [0] and Flintrock [1].
There are a few more tools listed on Spark Packages that target different cloud providers [2], too.
Disclaimer: I am the primary author of Flintrock and am a contributor to spark-ec2.
[0] https://github.com/amplab/spark-ec2
That said, I think `flintrock launch my-cluster` is almost as easy as doing `pip install ...`.
You do need an AWS account and you do need to set your preferences like region and key name in a config file, but I don't see how you can get out of doing even that without subscribing to some managed service like Databricks that abstracts everything away and replaces it with a nice Web UI.
> dcos package install spark
This gives you a Spark CLI, and installs the Spark Cluster Dispatcher so you can run jobs async. It also has options for installing the history server, and for SSL/Kerberos support.
Disclaimer: I work at Mesosphere and contribute to the DC/OS Spark package.
The new Spark DataFrames also make manipulating data almost as easy as with Python Pandas / R dplyr. The new Berkeley eDX course (https://courses.edx.org/courses/course-v1:BerkeleyX+CS105x+1...) is a very good explainer.
After Spark 2.0.0 is released, it wouldn't surprise me if it really takes off. (as long as setting up a cluster is a bit easier!)