Re:Apache Arrow 1.0
by cheez on Tuesday July 28, 2020 @02:14AM
(#23972804)
I just didn't trust the tools to do the job properly so for now I just split it up myself.
[ Parent
| on HN ]
Re:Apache Arrow 1.0
by mumblemumble on Tuesday July 28, 2020 @08:53AM
(#23975156)
Considering the number of segfaults I've witnessed that appear to originate in relatively recent versions of the Parquet library, I don't think your instincts are misplaced. I genuinely worry about data corruption.
I've been meaning to take a closer look at ORC. As a Spark user, I just sort of defaulted into Parquet. ORC is very similar, though, and seemingly gives every indication of being the more mature product.