Spark is built around a query planner that can aggregate multiple data sources and "push down" predicates, ordering and so on into the "leaf" node itself, much in the same way that Cockroach does. But I suspect that while this design is flexible and fast, to get the kind of OLTP-friendly performance that Cockroach is aiming for, a good the distributed execution logic has to be fine-tuned in a way that Spark -- being a general-purpose, backend-agnostic system -- cannot do. Spark can query things like Hive, Elasticsearch, Cassandra and so on; Cockroach has the luxury of only needing to target its own engine.

Also, last I checked Spark had very limited support for mutating data in SQL; it was designed for queries. Cockroach has to do both.