Re:We Need DevOps for ML Data
by seddonm1 on Tuesday April 28, 2020 @07:43PM
(#23013533)
If you are working within the Apache Spark ecosystem you can us DeltaLake https://delta.io/ to create 'merge' datasets which are transactional, versioned and allow time travel by both version number and timestamp.
[ Parent
| on HN ]
Re:We Need DevOps for ML Data
by jamesblonde on Wednesday April 29, 2020 @01:08AM
(#23015590)
Another alternative to Deltalake is Apache Hudi, which also includes bloom filters for indexing time-travel queries (efficiently exclude any files given the supplied time constraint). Z-ordered indexing in Deltalake is not available yet in open-source deltalake, only in Databricks version.