Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ...
by jnordwick on Wednesday April 19, 2017 @07:50PM
(#14152826)
So these ingested files are JSON or something semi-structured? I saw how the table was created, so maybe is was doing the schema-less key-value pair thing on table creation and running queries against that? I'm sure this is all in the docs somewhere, but while I have a lot of big db experience, I have about 2 hours of AWS.
[ Parent
| on HN ]
Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ...
by makmanalp on Wednesday April 19, 2017 @10:54PM
(#14153558)
So in this example the image says "lineitem_1t_parq" which makes me think that's probably parquet files (optimized and perhaps compressed columnar file format).
But it does the same stuff with CSVs too, perhaps a bit less efficiently. JSON is also definitely possible, but a bit more tricky because of the inherently nested structure and more complex format.
Take a look at this for the details on a similar idea: http://stratos.seas.harvard.edu/files/stratos/files/nodb-cac...
Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ...
by kakoni on Thursday April 20, 2017 @05:14AM
(#14155127)
That paper discusses PostgresRAW, which can be found here; https://github.com/HBPMedical/PostgresRAW
I wonder if there are any similar projects using psql as an engine?
Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ...
by makmanalp on Thursday April 20, 2017 @08:43AM
(#14156106)
I think it's pretty common - it's open source, nice and modular codebase, reasonably advanced execution engine with stuff like MVCC. But also redshift itself, citus, hadapt and greenplum are examples that come to my mind.