News for hackers. Stuff that matters.
Re:Giving meaning to 100B analytics events a day with Kafka,... by xstartup on Wednesday April 18, 2018 @11:41AM (#16868255)
Here is an example Pipeline which supports loading data from an unbounded source to BigQuery in batches using load jobs (evading BigQuery's Streaming Insert cost)

See: https://zero-master.github.io/posts/pub-sub-bigquery-beam/

Re:Giving meaning to 100B analytics events a day with Kafka,... by AWebOfBrown on Wednesday April 18, 2018 @12:40PM (#16868786)
Are you the author? OT but I'm amazed the author charged merely 100 euro for implementing that solution for the subject startup, even if they're cash-strapped. I'm not familiar with BigQuery, but I'm curious what a normal rate for solving that issue would look like.
Re:Giving meaning to 100B analytics events a day with Kafka,... by mentat on Wednesday April 18, 2018 @05:22PM (#16871313)
That's shockingly cheap for the value.
Re:Giving meaning to 100B analytics events a day with Kafka,... by manigandham on Wednesday April 18, 2018 @05:21PM (#16871303)
You can also skip pub/sub and/or use it to write files to cloud storage, then load from there by using a cloud function that will trigger the load job when a new object is created.
Re:Giving meaning to 100B analytics events a day with Kafka,... by cobookman on Wednesday April 18, 2018 @12:27PM (#16868670)
FYI dataflow's bigqueryio does not use stream inserts. Instead it batches the data into many load jobs
Re:Giving meaning to 100B analytics events a day with Kafka,... by oomkiller on Wednesday April 18, 2018 @01:47PM (#16869348)
That is configurable, but for unbounded PCollections it appears the default is to use streaming inserts https://beam.apache.org/documentation/sdks/javadoc/2.4.0/org...