News for hackers. Stuff that matters.

A comment on Apache Arrow 3.0. Parent comment

Re:Apache Arrow 3.0 by BadInformatics on Wednesday February 03, 2021 @05:42PM (#26020181)
HDF5 is pretty terrible as a wire format, so it's not a 1-1 comparison to Arrow. Generally people are not going to be saving Arrow data to disk either (though you can with the IPC format), but serializing to a more compact representation like Parquet.
[ Parent | on HN ]
Re:Apache Arrow 3.0 by gbrits on Wednesday February 03, 2021 @07:22PM (#26021069)
As I understand, arrow is particularly interesting since it’s wire format can be immediately queried/operated on without deserialization. Would saving an Arrow-structure as parquet not defeat that purpose, since your would need the costly deserialization step again on read? Honest question
[ Parent | on HN ]
Re:Apache Arrow 3.0 by BadInformatics on Wednesday February 03, 2021 @10:01PM (#26022100)
The FAQ [1] and this SO answer [2] explain it better than I can, but basically yes. However, the (de)serialization overhead is probably better than most alternative formats you could save to.

[1] https://arrow.apache.org/faq/ [2] https://stackoverflow.com/questions/56472727/difference-betw...

[ Parent | on HN ]