Re:Big data is dead (2023)
by ddalex on Monday May 27, 2024 @09:56AM
(#40490738)
Why is .parquet better than protobuf?
[ Parent
| on HN ]
Parquet is columnar storage, which is much faster for querying. And typically for protobuf you deserialize each row, which has a performance cost - you need to deserialize the whole message, and can't get just the field you want.
So, of you want to query a giant collection of protobufs, you end up reading and deserializing every record. For parquet, you get much closer to only reading what you need.
Thank you.
Re:Big data is dead (2023)
by nostrademons on Monday May 27, 2024 @04:45PM
(#40494518)
Parquet ~= Dremel, for those who are up on their Google stack.
Dremel was pretty revolutionary when it came out in 2006 - you could run ad-hoc analyses in seconds that previously would've taken a couple days of coding & execution time. Parquet is awesome for the same reasons.