It takes a fair amount of expertise to keep a spark cluster up and running. Maybe more people are using managed offerings like Databricks or Google Dataproc.