News for hackers. Stuff that matters.
Re:I Accidentally Saved My Company Half a Million Dollars by JohnMakin on Monday October 30, 2023 @01:28PM (#38072591)
This must be common or we worked at the same company, seen this exact pattern.

It went like:

App -> DynamoDB -> Kafka Connect Sink Process -> RDS -> Kafka

The reason for all the middle processes were because teams couldn't agree how to structure their data and the first app would dump literal nonsense sometimes so the Kafka connect process's job was to clean it and dump any of the nonsense they pumped into it. Pretty sure there was a gnarly log aggregation layer in the middle somewhere too IIRC.

There are no comments on this comment yet.