Most of this was Hadoop. Which executes mapreduce jobs written in sql on the jvm. The scalding jobs executed faster, except for one job for which we had to stick with Apache pig.

Spark jobs were native RDDs before the SQL Spark era.