Re:Reducing BigQuery Costs
by akira2501 on Monday February 05, 2024 @11:57PM
(#39270947)
> Rule #3. Letting "just anyone" run queries is as dangerous as casually handing a credit card to your drug-addicted cousin. Just wait until you get the bill!
And god help you if they figure out you can easily connect data studio to a BQ backend.
The core issue is data studio's date time time is incompatible with the BQ type so even if you've partitioned and sorted your dataset to the absolute maximum, the data studio query engine will ruin all that work and force a bunch of column conversions that require your entire dataset to be scanned again.
So, the conversation went:
"You need to edit all your data studio queries."
"Yea.. but there's like thousands of them across hundreds of our client facing 'dashboards.'"
"Okay, here's the bill, how many full time hours does one month of this insanity purchase? If you fix your queries this bill drops by 90%. FIX YOUR QUERIES!"
And god help you if they figure out you can easily connect data studio to a BQ backend.
The core issue is data studio's date time time is incompatible with the BQ type so even if you've partitioned and sorted your dataset to the absolute maximum, the data studio query engine will ruin all that work and force a bunch of column conversions that require your entire dataset to be scanned again.
So, the conversation went:
"You need to edit all your data studio queries."
"Yea.. but there's like thousands of them across hundreds of our client facing 'dashboards.'"
"Okay, here's the bill, how many full time hours does one month of this insanity purchase? If you fix your queries this bill drops by 90%. FIX YOUR QUERIES!"
They decided it was easier to leave GCP instead.