I've spent timing wiring up a similar setup with Airflow, and Celery on Kube, Kafka too, and running Spark clusters and echo the same sentiment. I also find Kubernetes easier/simpler compared to DC/OS, perhaps because it's a bit more structured and opinionated.