Re:A Data Pipeline Is a Materialized View
by glogla on Monday February 22, 2021 @06:23PM
(#26232020)
I do sadly have the opposite experience - "Yes, we are contractually obligated to give you data about every batch and everything we did with it - what do you mean excels with schema that differs day to day is not enough?"
...
The last step being computationally expensive kinda puts lower bound on latency of streaming replication from CDC, especially if you're trying to do it at scale and can't fine-tune each individual table partitioning.
Boy, how do I wish all data was incremental events.
...
The last step being computationally expensive kinda puts lower bound on latency of streaming replication from CDC, especially if you're trying to do it at scale and can't fine-tune each individual table partitioning.
Boy, how do I wish all data was incremental events.