Ask HN: How do I improve our data infrastructure?

Re:Ask HN: How do I improve our data infrastructure? by aeroevan on Saturday April 20, 2019 @10:44AM (#19706766)

It's 100GB compressed. Parquet does a very good job of compressing most data so that's where the estimate of 10x (so 1TB) uncompressed was mentioned as a rule of thumb.

Parquet also supports much better access mechanisms, like being able to deserialize a single column without having to read in entire rows.

But like you mentioned, 1TB of data in a traditional database isn't that bad.

[ Parent | on HN ]

Re:Ask HN: How do I improve our data infrastructure? by usgroup on Saturday April 20, 2019 @10:56AM (#19706828)

... also remembering that a traditional dB will typically not store data raw. Row compression is normal and disk compression is normal . The typical column store advantage is block compression, predicate pushdown and column order storage.

[ Parent | on HN ]

Re:Ask HN: How do I improve our data infrastructure? by sixdimensional on Saturday April 20, 2019 @12:58PM (#19707558)

Regular databases such as SQL Server and Oracle have had columnar compression built in as an option along with the row stores for years now. I use it in SQL Server a lot and it works great.

[ Parent | on HN ]

Re:Ask HN: How do I improve our data infrastructure? by riku_iki on Saturday April 20, 2019 @10:58AM (#19706844)

you can run sql DB over compressed filesystem, and some DBs allow you to compress tables too

> like being able to deserialize a single column without having to read in entire rows.

and it reads filesystem's whole page anyway

[ Parent | on HN ]

Re:Ask HN: How do I improve our data infrastructure? by aeroevan on Monday April 22, 2019 @12:31PM (#19720269)

Sorry for the late reply, but parquet is a columnar format so if it's big enough data, you should have multiple pages/blocks of data in a single column for a specific row group, and then be able to seek to the next row group and sequentially read the next set of blocks.

[ Parent | on HN ]