Edit: Wait, did you imply that the SQL you replaced with Scala ultimately gets transpiled into SQL (spark or other dataframe engine) and "pushed down" to the database? Now you're just maintaining Scala and not a rats nest of SQL strings concatenated together? If so, that's pretty awesome.
The SQL was HiveQL. Spark didn't really support SQL at the time and was native JVM Scala RDDs.
HiveQL with UDFs compiled to JVM bytecode typically ended up performing worse than scalding that corresponds to straight raw jvm map reduce jobs that are the real native interface of Hadoop.
The real performance drag from HQL came from developers writing inefficient map reduce jobs with extra joins to avoid writing Java UDFs because they were a huge PITA.
In scalding, developers wrote map reduce jobs with as few group bys as possible. Our hadoop infra team actually complimented us on how great the execution profile of our teams jobs were.
Spark jobs were native RDDs before the SQL Spark era.
Got rid of 1000s of lines of Java and SQL for 100s of lines of scala, scalding and Spark.
The codebase size on the first migration was 10% of the original Java+SQL. overall, a team of 6 worked for 4 years to produce 10K lines of high quality Scala code for a very complex app and platform after the first transition from 10K lines of java+sql to 1K lines of scala.
Never experienced something like this before or after.
At the next place I wrote 400 lines of C++ mapreduce over 3 days, which could have been 40 lines of sql or 10 lines of scalding/spark.
Jarring transition.