News for hackers. Stuff that matters.
Re:Introducing Spark Datasets by JoshRosen on Monday January 11, 2016 @01:43PM (#10882231)
The blog is referring to source- and binary-API compatibility guarantees that Spark provides.

IIRC, in order for DataFrames and Datasets to share a common parent class certain common methods' return types would need to change in a binary-incompatible way that would break DataFrame code which was compiled against older Spark versions.

Due to Spark's API compatibility and versioning policies, this type of compatibility break can only occur in major releases, so it can only be done in Spark 2.0.0 and not Spark 1.6.

There are no comments on this comment yet.