News for hackers. Stuff that matters.
Re:Datastream for BigQuery Preview by atwebb on Friday September 16, 2022 @01:34PM (#32869422)
Yeah the Debezium connectors have some issues that really get in the way. I'm less familiar with BQ but some other DBs, the data typing is really, really basic

(case when string then varchar(4000/max))

and similar. It looks like a relatively easy thing to incrementally improve.

Re:Datastream for BigQuery Preview by gunnarmorling on Friday September 16, 2022 @02:31PM (#32870177)
Hey, Gunnar here from the Debezium team. I would love to learn more about those issues you encountered with Debezium. Perhaps it's something we could improve. So if you could share your feedback either here, on our mailing list (https://groups.google.com/g/debezium), that would be awesome. Thanks!
Re:Datastream for BigQuery Preview by kppullin on Friday September 16, 2022 @04:37PM (#32871710)
For us, debezium has been working rather well in recent memory (there were a couple PRs I submitted that I'm proud of, even if tiny!). Most of the issues are on the kafka connector side, whether it's the bigquery sink, jdbc sink, and s3 sink.

A couple things that do pop to mind as it relates to debezium include better capabilities around backup + restores and disaster recovery, and any further hints around schema changes. Admittedly I haven't looked at these areas for 6+ months so they may be improved.

Re:Datastream for BigQuery Preview by atwebb on Monday September 19, 2022 @02:20PM (#32901924)
Well I'm trying to find it now! Maybe it has been updated. It was essentially a case statement on the destinations which went for some wide data types but, looking now, I see where it is looking at the schemas and getting more precision.