Re:Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Goog...
by johnnunn on Sunday September 25, 2022 @12:03AM
(#32968843)
I have a use case, where my company's application logs will be shipped to S3 in a directory structure such as application/timestamp(one_hour)_logs.parquet. We want to build a simple developer focussed UI, where we can query for a given application for a time range and retrieve a bunch of s3 blobs in that time range and brute force search for the desired string. I see that roapi offers a REST interface for a fixed set of files but I would like to dynamically glob newer files. Are there are alternatives that can be used too ? Thanks
[ Parent
| on HN ]
Re:Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Goog...
by mkane on Sunday September 25, 2022 @07:17AM
(#32970372)
If you're already using parquet, it might be worth looking at the concept of datasets e.g. https://arrow.apache.org/docs/python/generated/pyarrow.parqu...
Re:Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Goog...
by cube2222 on Sunday September 25, 2022 @07:23AM
(#32970400)
Amazon Athena + AWS Glue for schema discovery can do this.
Re:Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Goog...
by marek_leisk2 on Sunday September 25, 2022 @05:55AM
(#32970072)
Trino can do this.