News for hackers. Stuff that matters.

A comment on Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 Data. Parent comment

Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ... by makmanalp on Wednesday April 19, 2017 @10:54PM (#14153558)
So in this example the image says "lineitem_1t_parq" which makes me think that's probably parquet files (optimized and perhaps compressed columnar file format).

But it does the same stuff with CSVs too, perhaps a bit less efficiently. JSON is also definitely possible, but a bit more tricky because of the inherently nested structure and more complex format.

Take a look at this for the details on a similar idea: http://stratos.seas.harvard.edu/files/stratos/files/nodb-cac...

[ Parent | on HN ]
Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ... by kakoni on Thursday April 20, 2017 @05:14AM (#14155127)
That paper discusses PostgresRAW, which can be found here; https://github.com/HBPMedical/PostgresRAW

I wonder if there are any similar projects using psql as an engine?

[ Parent | on HN ]
Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ... by makmanalp on Thursday April 20, 2017 @08:43AM (#14156106)
I think it's pretty common - it's open source, nice and modular codebase, reasonably advanced execution engine with stuff like MVCC. But also redshift itself, citus, hadapt and greenplum are examples that come to my mind.
[ Parent | on HN ]