Re:Scalding at Etsy
by StefanKarpinski on Tuesday March 04, 2014 @07:18PM
(#7344282)
At that point you're just implementing a crappy statically type language inside of a Ruby DSL. Using a real statically typed JVM language that's more expressive than Java – i.e. Scala – would be much better. So, that's basically Scalding. Of course, Scalding wasn't even close to existing when we chose Cascading.JRuby, so yeah.
Yeah I'm just talking counterfactually. At the time there were no solid cascading DSLs whatsoever, cascalog wasn't even around for another 2 years or so iirc. Cascading itself was the new hotness vs writing raw map reduce, so c.j was kind of on the edge of what was out there. The decision to keep UDFs outside of Ruby was made at the outset (out of concern for performance, and being icing on the cake, etc.) and not really revisited.
I think the thing to remember with the c.j stuff is at the time Cascading's Java API was pretty much state of the art, and we had to shore up a lot of what was in c.j in order to get to a viable DSL. And at that point, it was a huge leap forward from a readability standpoint. (Cascading's API had the same shortcomings as our c.j API wrt types etc, fwiw)