JoshRosen
Unfortunately, the project has been largely abandoned and the current version in the Netflix GitHub repository won't work out of the box for many companies due to lack of support for new AWS regions, instances, reservation types, and services, as well as showstopping bugs (such as https://github.com/Netflix/ice/issues/210). In order to effectively use ICE today most users will need to maintain their own patched fork.
I'm currently campaigning to create a community-maintained version of ICE, with committers from multiple organizations, in order to revive the project: https://github.com/Netflix/ice/issues/240#issuecomment-29960...
IIRC, in order for DataFrames and Datasets to share a common parent class certain common methods' return types would need to change in a binary-incompatible way that would break DataFrame code which was compiled against older Spark versions.
Due to Spark's API compatibility and versioning policies, this type of compatibility break can only occur in major releases, so it can only be done in Spark 2.0.0 and not Spark 1.6.
https://issues.apache.org/jira/browse/SPARK-2360 https://github.com/apache/spark/pull/1351