At the time Spark was started, Java didn't have lambdas or closures, making the equivalent Java code for functional-programming-style ML code much more verbose. Whether the same decision would be made today is an open question (and probably depends on who you ask). A lot of other frameworks have made different decisions, such as keeping the core of the project in Java and supporting other languages as shims on top (in my opinion a more sensible decision...)