Parquet: More than just “Turbo CSV”

Re:Parquet: More than just “Turbo CSV” by julik on Tuesday April 04, 2023 @07:38AM (#35438029)

There is a point which one needs to be aware of with Parquet: if you are not stepping into it with tools well suited for it (that is: Go, C++, Java or Python codebase/runtime, mainstream platform...) you are going to have a bad time. For all its advantages, Parquet is _very_ complicated (IMO overcomplicated, on multiple fronts), and uses binary dependencies which you absolutely need to carry around (Thrift?..). Up to a certain data size the advantage of CSV is not that it is a better format (it is decidedly, absolutely worse/more ambiguous than Parquet) but the fact that you can write a valid CSV in a couple of lines of anything, and read in a couple of lines of anything. If you ask me, Parquet - for all its advantages - is a product of the Cambrian explosion of the Apache-big-data-projects. Very little in its design warrants (or requires) using things like Thrift, so I won't exclude that something less overcomplicated (and more welcoming, and requiring less dependencies) will appear on the scene at some point.

[ Parent | on HN ]

Re:Parquet: More than just “Turbo CSV” by cm2187 on Tuesday April 04, 2023 @09:28AM (#35439248)

To be honest any file format that would do 1) strong typing/binary data, 2) string deduplication would capture most of the advantages of parquet, except being a column store format which helps in some cases and hurts in others.

[ Parent | on HN ]

Re:Parquet: More than just “Turbo CSV” by julik on Tuesday April 04, 2023 @05:33PM (#35446127)

And even using a columnar format but without all the complexity of Parquet would be comparatively easy (when writing using buffers)

[ Parent | on HN ]

Re:Parquet: More than just “Turbo CSV” by nomadluap on Tuesday April 04, 2023 @07:22PM (#35447183)

At that point just start using an SQLite database as a data format.

[ Parent | on HN ]

Re:Parquet: More than just “Turbo CSV” by jonstewart on Tuesday April 04, 2023 @08:32PM (#35447701)

SQLite is a row-oriented store, not a column-oriented store, ie, there are fundamental tradeoffs between the two.

[ Parent | on HN ]

Re:Parquet: More than just “Turbo CSV” by cm2187 on Tuesday April 04, 2023 @10:21PM (#35448489)

But sqlite has no compression by default.

[ Parent | on HN ]

Re:Parquet: More than just “Turbo CSV” by bsder on Tuesday April 04, 2023 @05:16PM (#35445947)

In addition, very few of these kind of serialization formats are useful for mutable data.

One thing that I have been struggling with is to find an in-memory, MUTABLE data format/data structure that can be used between different processes in shared memory. I'm not sure one exists.

[ Parent | on HN ]

Re:Parquet: More than just “Turbo CSV” by julik on Tuesday April 04, 2023 @05:36PM (#35446160)

I have been using SQLite DBs to that end with some amount of success. If I remember well a lot of software of yore used BDB to the same effect.

[ Parent | on HN ]

Re:Parquet: More than just “Turbo CSV” by deepsun on Thursday April 06, 2023 @02:05AM (#35464742)

Apache Arrow?

As far as I understand it, it's more for cross-process in-memory fast access (e.g. numbers crunching), but it's mutable.

[ Parent | on HN ]

Re:Parquet: More than just “Turbo CSV” by bsder on Thursday April 06, 2023 @06:30PM (#35475408)

As far as I can tell, Apache Arrow is not very mutable. It basically has to reallocate the universe.

I don't need large data between processes. I need short-lived medium data ... large enough and short-lived enough to make serialization painful but mutable enough to make "stores" kind of annoying.

[ Parent | on HN ]

Re:Parquet: More than just “Turbo CSV” by manigandham on Sunday April 16, 2023 @06:37AM (#35588814)

Use Redis. It's an in-memory data structure server, first created to have the same data structures usable and accessible across processes and languages.

Or plenty of other in-memory database products (Apache Ignite) or fast key/value (ScyllaDB) or just use RDBMS (Postgres or SQLite).

[ Parent | on HN ]

Re:Parquet: More than just “Turbo CSV” by imtringued on Wednesday April 05, 2023 @02:17AM (#35450060)

>but the fact that you can write a valid CSV in a couple of lines of anything, and read in a couple of lines of anything.

I seriously hope you are implying the usage of a library.

Hand rolled CSV implentations are generally wrong and ruin your data.

[ Parent | on HN ]

Re:Parquet: More than just “Turbo CSV” by lisasays on Wednesday April 05, 2023 @09:38PM (#35462996)

The beauty of CSV is that you don't -need- a library to access the data.

It's just there, right in front of you.

Hand rolled CSV implentations are generally wrong and ruin your data.

Yeah sure - asymptotically. Once you've gotten a handle on your data, you can take care of all the bulletproofing later.

None of this is meant to be cheeky, BTW. The learning curve and general cognitive overhead of Parquet is very considerable.

[ Parent | on HN ]