Re:Simdjson – Parsing Gigabytes of JSON per Second
by teej on Thursday February 21, 2019 @02:19AM
(#19215168)
You’ve kinda just described AWS Athena.
[ Parent
| on HN ]
Re:Simdjson – Parsing Gigabytes of JSON per Second
by Cthulhu_ on Thursday February 21, 2019 @03:43AM
(#19215462)
This comment needs to be higher up; Amazon has a service for doing just this, dumping 'dumb' files (like json, csv, etc) into S3 buckets and performing SQL queries on them. No need to have to think about how to store things for future querying.
Re:Simdjson – Parsing Gigabytes of JSON per Second
by owenmarshall on Thursday February 21, 2019 @11:34AM
(#19217991)
I've used Athena really effectively to solve similar problems. If your data storage is relatively small and/or your queries relatively infrequent, JSON can be a good fit. As one of those dimensions expands, you can decrease costs/increase performance by converting to Parquet and compressing.