Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ...
by makmanalp on Wednesday April 19, 2017 @05:58PM
(#14152183)
So this is a good question - the answer is that you're comparing apples to oranges. This is all about in-situ data processing, which basically means that you can run reasonably efficient, optimized queries on just regular old files that you have lying around, without having to do anything special. This is as opposed to a situation where you've spent a ton of time, effort and expense actually ingesting the data into whatever data system you're running (in this case redshift). So it's usually said that these systems are good for ad-hoc queries, i.e. one-off queries on files and datasets that you want to explore more, but it doesn't make sense to invest the hours upon hours of waiting and storage / cpu resources to bring them into your database just to make a few queries.
[ Parent
| on HN ]
Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ...
by jnordwick on Wednesday April 19, 2017 @07:50PM
(#14152826)
So these ingested files are JSON or something semi-structured? I saw how the table was created, so maybe is was doing the schema-less key-value pair thing on table creation and running queries against that? I'm sure this is all in the docs somewhere, but while I have a lot of big db experience, I have about 2 hours of AWS.
Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ...
by makmanalp on Wednesday April 19, 2017 @10:54PM
(#14153558)
So in this example the image says "lineitem_1t_parq" which makes me think that's probably parquet files (optimized and perhaps compressed columnar file format).
But it does the same stuff with CSVs too, perhaps a bit less efficiently. JSON is also definitely possible, but a bit more tricky because of the inherently nested structure and more complex format.
Take a look at this for the details on a similar idea: http://stratos.seas.harvard.edu/files/stratos/files/nodb-cac...
Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ...
by kakoni on Thursday April 20, 2017 @05:14AM
(#14155127)
That paper discusses PostgresRAW, which can be found here; https://github.com/HBPMedical/PostgresRAW
I wonder if there are any similar projects using psql as an engine?
Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ...
by makmanalp on Thursday April 20, 2017 @08:43AM
(#14156106)
I think it's pretty common - it's open source, nice and modular codebase, reasonably advanced execution engine with stuff like MVCC. But also redshift itself, citus, hadapt and greenplum are examples that come to my mind.