News for hackers. Stuff that matters.
Re:RDDs are the new bytecode of Apache Spark by pacala on Sunday May 31, 2015 @12:11AM (#9633035)
It's software, everything is possible :) It is fairly straightforward to add compile time information via a simple case class definition. There is still a dynamic type check that json matches the case class, but the spec of the structure of the data is done in one place, not spread out arbitrarily over the query code base.
Re:RDDs are the new bytecode of Apache Spark by rxin on Sunday May 31, 2015 @01:44AM (#9633229)
That part is actually coming: https://github.com/apache/spark/pull/5713
Re:RDDs are the new bytecode of Apache Spark by threeseed on Sunday May 31, 2015 @01:08AM (#9633173)
Dynamic type checks are actually pretty damn useful.

For example the MongoDB adapter samples the data at runtime to determine what the DataFrame definition i.e. types is.