Ask HN: How do I improve our data infrastructure?

Re:Ask HN: How do I improve our data infrastructure? by sologoub on Saturday April 20, 2019 @12:15PM (#19707304)

That’s the beauty of BQ - it scales well, but it works just fine in smaller use cases. It doesn’t get simpler than SQL.

Another item to consider is that BQ now has ML (simpler) models built in, further reducing the complexity of your pipeline: https://cloud.google.com/bigquery/docs/bigqueryml-intro

If you are not on GCP, then I’d consider AWS Athena for querying the parquet files, but you still have to structure these efficiently beforehand.

[ Parent | on HN ]

Re:Ask HN: How do I improve our data infrastructure? by remilouf on Sunday April 21, 2019 @03:30AM (#19710967)

I will consider that. How about Redshift?

[ Parent | on HN ]

Re:Ask HN: How do I improve our data infrastructure? by aarbor989 on Sunday April 21, 2019 @12:56PM (#19713054)

We had Redshift for our 23TB+ dataset and it worked great. The downside is it can get pricy, so do a cost analysis before you commit. Also know that views in redshift are not materialized so it’s more efficient to create physical tables of the views - which then adds maintenance overhead. The last thing I’ll add is that you’ll need to experiment with compression settings for your data. For us, a combination of ZSTD and bytedict was all we needed

[ Parent | on HN ]