TileDB: Storing massive dense and sparse multi-dimensional array data

Re:TileDB: Storing massive dense and sparse multi-dimensiona... by ankitrohatgi on Thursday October 26, 2017 @11:26AM (#15559697)

I am curious about how the query performance compares to working with JSON files in Spark for ~100GB data.

[ Parent | on HN ]

Re:TileDB: Storing massive dense and sparse multi-dimensiona... by jakebol on Thursday October 26, 2017 @01:23PM (#15561023)

Jake from TileDB, Inc. here: Depending on the structure of the JSON files you are querying you maybe able to take advantage of columnar compression and massively reduce the dataset size (especially if the json files contain numeric data). Also, repeat queries will not have to re-parse the JSON files. This may speed up queries quite a lot, but it depends on the specifics of your problem.

[ Parent | on HN ]

Re:TileDB: Storing massive dense and sparse multi-dimensiona... by StanSeltser on Thursday October 26, 2017 @11:08PM (#15565386)

Stanislav Seltser, Petacube you are talking comparing structured workload(array-based TileDB) to unstructured one (JSON+Spark). Once you convert your JSON to sparce array structure (one time conversion) TileDB will beat Spark+JSON by several orders of magnitude. Caveat: assuming your spark+json workoad is a some heayy processing not a lightweight one.

[ Parent | on HN ]

Re:TileDB: Storing massive dense and sparse multi-dimensiona... by maxpert on Thursday October 26, 2017 @11:28AM (#15559728)

They have some benchmarks in paper https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17...

[ Parent | on HN ]