News for hackers. Stuff that matters.

A comment on We built a Modern Data Stack from scratch and reduced our bill by 70%. Parent comment

Re:We built a Modern Data Stack from scratch and reduced our... by jchandra on Sunday March 09, 2025 @02:44PM (#43312290)
As for BigQuery, while it's a great tool, we faced challenges with high-volume, small queries where costs became unpredictable as it is priced per data volume scanned. Clustered tables, Materialised views helped to some extent, but they didn’t fully mitigate the overhead for our specific workloads. There are ways to overcome and optimize it for sure so i wouldn't exactly put it on GBQ or any limitations.

It’s always a trade-off, and we made the call that best fit our scale, workloads, and long-term plans

[ Parent | on HN ]
Re:We built a Modern Data Stack from scratch and reduced our... by vivahir215 on Sunday March 09, 2025 @02:49PM (#43312335)
Hmm, Okay.

I am not sure if managing kafka connect cluster in too expensive in long term. This solution might work for you based on your needs. I would suggest to look for alternatives.

[ Parent | on HN ]
Re:We built a Modern Data Stack from scratch and reduced our... by throwaway7783 on Sunday March 09, 2025 @05:33PM (#43314110)
Did you consider slots based pricing model for BQ?
[ Parent | on HN ]