News for hackers. Stuff that matters.
Re:Apache Arrow and MinIO by anderspitman on Sunday September 20, 2020 @07:28PM (#24538311)
Is Arrow data commonly written to disk? I was under the impression (as someone who doesn't use it currently) it's usually converted to Parquet for this purpose.
Re:Apache Arrow and MinIO by andygrove on Sunday September 20, 2020 @09:00PM (#24538788)
Arrow is a memory format and optimized for efficient vectorized processing in memory. Although it is possible to persist the Arrow format to disk, it is more common to use Parquet.
Re:Apache Arrow and MinIO by thundergolfer on Sunday September 20, 2020 @08:24PM (#24538599)
I'd be interested to know any benefits of converting to Parquet if you know your reader supports Arrow.
Re:Apache Arrow and MinIO by lmeyerov on Sunday September 20, 2020 @09:53PM (#24539030)
Compression. For our app, we do multilevel caches, where warm is arrow (in-memory, inflight, streaming, ...) and cold (persistent/scaleout) is parquet (ssd, s3, ...).