Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

Re:Show HN: s3-lambda – Lambda functions over S3 objects: ea... by hayd on Sunday February 12, 2017 @03:39PM (#13630619)

see also aws athena https://aws.amazon.com/athena/ ?

[ Parent | on HN ]

Re:Show HN: s3-lambda – Lambda functions over S3 objects: ea... by dajohnson89 on Sunday February 12, 2017 @09:23PM (#13632693)

That seems cool but paying per query (per TB scanned) frightens me. I imagine having to fret about how efficient my queries are...

[ Parent | on HN ]

Re:Show HN: s3-lambda – Lambda functions over S3 objects: ea... by illumin8 on Monday February 13, 2017 @01:54AM (#13633581)

It's not that bad. You can compress the data on S3 in ORC or Parquet format, and you only pay for the compressed data you read, so 1TB can be 130GB after compression. Plus, these formats store summary data, so queries like SELECT COUNT don't have to do a full table scan - they can read just a few KB of summary data for the result.

[ Parent | on HN ]

Re:Show HN: s3-lambda – Lambda functions over S3 objects: ea... by dajohnson89 on Sunday February 19, 2017 @02:54PM (#13681624)

But that's a lot of work....Just to have sane costs for reads of your data

[ Parent | on HN ]

Re:Show HN: s3-lambda – Lambda functions over S3 objects: ea... by illumin8 on Wednesday March 01, 2017 @02:43PM (#13767507)

It's actually just two commands:

1. hive 2. INSERT INTO parquet_table SELECT * FROM csv_table;

[ Parent | on HN ]

Re:Show HN: s3-lambda – Lambda functions over S3 objects: ea... by wellsjohnston on Sunday February 12, 2017 @03:59PM (#13630760)

I did not know about this...looks like Amazon's version of BigQuery. Fantastic!

[ Parent | on HN ]

Re:Show HN: s3-lambda – Lambda functions over S3 objects: ea... by nstj on Sunday February 12, 2017 @08:49PM (#13632539)

Somehow I'd overlooked this too: nice find

[ Parent | on HN ]