News for hackers. Stuff that matters.
Re:Introducing Spark Datasets by rkrzr on Monday January 11, 2016 @05:04AM (#10879897)
Does somebody know why these improvements could not be integrated into DataFrames directly?

Why add another very similar layer of abstraction?

They mention:

"Unification of DataFrames with Datasets – due to compatibility guarantees, DataFrames and Datasets currently cannot share a common parent class. With Spark 2.0, we will be able to unify these abstractions with minor changes to the API, making it easy to build libraries that work with both."

Anybody know what those "compatibility guarantees" are?

Re:Introducing Spark Datasets by nl on Monday January 11, 2016 @06:51AM (#10880256)
Dataframes aren't strongly typed (while RDDs are).

It's difficult to add strong typing to an existing API.

I certainly prefer using DataFrames generally, but sometimes you need to switch to RDDs because some operations are easier.

Hopefully Datasets bridge the gap well. I haven't used them yet, but the examples look nice.

Re:Introducing Spark Datasets by JoshRosen on Monday January 11, 2016 @01:43PM (#10882231)
The blog is referring to source- and binary-API compatibility guarantees that Spark provides.

IIRC, in order for DataFrames and Datasets to share a common parent class certain common methods' return types would need to change in a binary-incompatible way that would break DataFrame code which was compiled against older Spark versions.

Due to Spark's API compatibility and versioning policies, this type of compatibility break can only occur in major releases, so it can only be done in Spark 2.0.0 and not Spark 1.6.