Do you need to run this pipeline more than once a day? Is it sufficiently important for your business case? I feel like 25 is a very small sum if any of them are remotely true.
On a related note, how practically reliable is the DataPrep->DataFlow workflow? Could a reasonably smart analyst with zero programming experience set them up and run them? Feel like that's who this workflow is built for, but I've not had the best of experiences with Trifacta demos in AWS with reliability.
I agree a warehouse could get more expensive, but that’s where Athena or BigQuery is supposed to come in? BigQuery is rather expensive though. It can’t support reading Parquet natively (of course Google insists on their own format) so you have to pay their heavy tax versus just object storage and elastic compute.
Also the classic BigQuery UI is way less buggy than the new one, and the PMs won’t take bugs for the new UI.
“ Our benchmark results show that Apache Beam has a noticeable impact on the performance of DSPSs in almost all cases. Programs developed using Apache Beam suffered from a slowdown of up to a factor of 58 in the worst case. At the same time, there is one scenario where the query developed using Apache Beam is about as fast as its counterparts using the APIs of the corresponding DSPS. However, for most scenarios we observed a slowdown of at least a factor three.”
Also DataPrep was amazing until you run the DataFlow pipeline it produces and a < 1TB dataset cleaning costs $25...
Have had good experiences once the data is in bigquery and bq keeps costs low. But lately Ive been trying to figure out a semi-managed way of doing exactly-once stream processing for cheaper than dataflow. Possibly an architecture around cloud run might work.