Re:How ksqlDB Works: Internal Architecture and Advanced Feat...
by mjdrogalis on Thursday September 02, 2021 @04:07PM
(#28396722)
Is it an option to model it as a table/table join? t/t triggers when there's a change on either side of the expression.
If that doesn't work, feel free to swing by our community Slack room and we can get into the weeds. :)
Re:How ksqlDB Works: Internal Architecture and Advanced Feat...
by arman_ashrafian on Thursday September 02, 2021 @04:47PM
(#28397154)
Oh cool need to think about this a bit more. I’ll see you in the slack channel :)
I'm just going to throw out another problem since you guys are here :)
The current batch process joins some tables before sending data to elasticsearch. This means that the debezium connector doesn't write all the data I need into kafka. I was thinking I could create a materialized table in ksql with infinite retention for the other postgres tables I need to join on. Then when I stream in an update for the data I want in elasticsearch, I can join on these tables.
The issue is that a Stream-Table join only gets triggered when the stream changes. This means that when the data in the tables change we will not see these updates in elasticsearch.
I guess my only option is to join everything in our app and then produce the full message to the elastic sink topic?