Re:Giving meaning to 100B analytics events a day with Kafka,...
by AWebOfBrown on Wednesday April 18, 2018 @12:40PM
(#16868786)
Are you the author? OT but I'm amazed the author charged merely 100 euro for implementing that solution for the subject startup, even if they're cash-strapped. I'm not familiar with BigQuery, but I'm curious what a normal rate for solving that issue would look like.
Re:Giving meaning to 100B analytics events a day with Kafka,...
by mentat on Wednesday April 18, 2018 @05:22PM
(#16871313)
That's shockingly cheap for the value.
Re:Giving meaning to 100B analytics events a day with Kafka,...
by manigandham on Wednesday April 18, 2018 @05:21PM
(#16871303)
You can also skip pub/sub and/or use it to write files to cloud storage, then load from there by using a cloud function that will trigger the load job when a new object is created.
Re:Giving meaning to 100B analytics events a day with Kafka,...
by cobookman on Wednesday April 18, 2018 @12:27PM
(#16868670)
FYI dataflow's bigqueryio does not use stream inserts. Instead it batches the data into many load jobs
Re:Giving meaning to 100B analytics events a day with Kafka,...
by oomkiller on Wednesday April 18, 2018 @01:47PM
(#16869348)
That is configurable, but for unbounded PCollections it appears the default is to use streaming inserts https://beam.apache.org/documentation/sdks/javadoc/2.4.0/org...
See: https://zero-master.github.io/posts/pub-sub-bigquery-beam/