nathairtras
Have grown to really appreciate Spark in the Hadoop space. Started with plans to go with Impala, then went to Hive due to stability concerns, and finally to Spark due to speed / flexibility. You can write code against a data frame, or write Spark SQL, so you still have SQL.
HDFS has benefits over other storage approaches, if you are running Spark in the same cluster you get data proximity. But you can go with a different storage back-end. That costs in performance. "Performance of multiple query and enrichment jobs concurrently executed resulted in 90% longer execution times." https://redhatstorage.redhat.com/2018/06/25/why-spark-on-cep...
Unless you really have BIG data, you're invoking a lot of maintenance overhead to support a cluster when you may do just fine without.
Haven't had the freedom to explore other possibilities until recently, very interested in how Spark on k8s is working out. (Same comment could be made here as above and elsewhere - do you really need k8s? But I want to play with k8s and learn more about it, so... for that purpose I 'need' it.) https://spark.apache.org/docs/latest/running-on-kubernetes.h...
And there's always the cloud route. You can run an EMR job that uses files in s3. There is a cost, but you do not need to support a cluster in the same way. Or if you're feeling adventurous, use Lambda. https://www.qubole.com/blog/spark-on-aws-lambda/
And Spark isn't the only option. Have started learning about Dask, also looks very interesting for performing some of the same tasks. https://dask.org
Nest products were the one exception. Previously owned several Nest Protect units and a thermostat. Moved, and installed a Nest thermostat and some temperature sensors. Just a couple of weeks ago, finished writing a data collection script to look at the temperatures in various rooms of the house. Mostly to have some real-world personal data to play with. Was hoping to write an Alfred workflow to change the temperature from my laptop.
Now I've lost those options, and in addition, I will have to have a Google account. (That is how the notice reads, right? You don't have to convert immediately, but you will have to convert eventually?)
Almost consider it my fault. I trusted Google in this one case after losing trust in Google in many other cases. “Fool me once, shame on — shame on you. Fool me — you can't get fooled again.”
Will probably start looking for alternatives. Hopefully the corporate machine is paying attention to this, and will realize that attempting to force users into your ecosystem will drive many of them to different ecosystems. But it will take time and effort to regain user trust.