One benefit of using the table as a source of truth this way is that you can run migrations on the entire “stream” if you will, and republish the events in order, in the new schema definition.
Another benefit is that you can run SQL queries against your entire transaction history. Some (imo sketchy) tools exist that try to give you a SQL interface to Kafka, but that’s just bad. It’s much easier to have your cake and eat it in postgres than it is in Kafka, imo. Unless your data volume is absurd, in which case Kafka is your only realistic choice.
Of course, in a “literal” sense you’re correct, it’s not possible. But from a practical point of view, if your bit empor all data implementation is stable, you can totally depend on your ability to produce streams from a database. We do this at my current place of work all the time.
https://engineering.linkedin.com/distributed-systems/log-wha...
Pointing out (GP and yourself) that this is an 'asymmetric duality' is informative and valid.
A symmetric dual is one where there is equivalence: the dual of the dual is the primal.
But it is possible to have a "weak" duality that is assymmetric, where the dual encompasses the primal -- where there are regions of equivalence instead of a 1-to-1 equivalence.
I'm not sure if stream-duality fits this conceptual mold exactly but thank you for providing a counterpoint to my assertion. I'm going to think on it.
Also, that you believing it is the most profitable mindset - for them. Company blogs are marketing, first and foremost.
I don’t have a proof, but For things like state transitions, I’ve often solved this with a <new_state>_at column. Seeing that field populated would indicate a second event need be emitted. (Assumes no cycles in the state machine.)
Often that table is not the one you have, even if (perhaps especially if) you emitted events for every update — seems like some entropy-like force pushes the universe toward emitting details in events that aren’t reflected in the table records, over a long enough timeline.
Difference algebra over the deltas could find some interesting relevance.
That one would so confidently assert that "Codata is the categorical dual of Data" has me thinking there might be something here?
Anyone want to tell me how far off I am?
More usefully, data is a finite data type and codata is an infinite data type.
Some example:
sum = foldr (+) 0
sum [0,1,2,3]
> 6
sum [0..]
...loops forever...
sum' = scanl (+) 0
sum' [0,1,2,3]
> [0, 1, 3, 6]
sum' [0..]
> [0, 1, 3, 6, 10...
Induction describes how to process data without diverging - process each input element in a finite time.
Co-induction describes how to process codata without diverging - produce each output element in a finite time.
This is useful if you want to prove things about infinite processes like operating systems.
You can construct a state (table) from a stream, but you cannot do the reverse. You cannot deconstruct a table into its original stream because the WAL / CDC information is lost -- you can only create a new stream from the table. This means you lose all ability to retrieve previous states by replaying the stream. Information is lost.
Duality in math is an overloaded term but it generally means you can go in either direction. This is not true here.