News for hackers. Stuff that matters.
Re:The Reactive Monolith – How to Move from CRUD to Event So... by 0xFACEFEED on Wednesday September 29, 2021 @10:07PM (#28702432)
I feel bad for anyone who got sucked into event sourcing for general purpose applications. Like... I'm burning a candle for you right now.

The event stream stuff makes sense in some specialized use cases (LMAX?) but for general purpose stuff? Oof, no.

EDIT: Also, don't hate me folks, but any article that references Fowler should be looked at with just a little bit of suspicion. Not hating the guy but he's blown quite a bit of hot air in his time.

Re:The Reactive Monolith – How to Move from CRUD to Event So... by goostavos on Thursday September 30, 2021 @02:33AM (#28703760)
Count me in the long line of suckers :(

Probably one of the largest technical blunders of my career (or at least the largest so far!) was pulling a team down into ES hell on a greenfield project.

Turns out, growing an event sourced application is fantastically hard[0]. Ours eventually became a twisted opaque mess which was terrifying to modify due to (a) the sheer volume of code it took to do so, and (b) eventually losing control over who read which events and where as the team grew.

[0] https://chriskiehl.com/article/event-sourcing-is-hard

Re:The Reactive Monolith – How to Move from CRUD to Event So... by beaker52 on Thursday September 30, 2021 @03:17AM (#28703942)
I'm working with a team that's in the process of decomissioning a previous teams attempt at ES. Even the new team wanted to do ES, whilst staring at the remains of the previous attempt. Luckily they haven't.

Usually arguments go "we need an audit log so ES is a natural fit and we'll be able to iterate quicker because we can just re-project when we get things wrong, oh and testing is really straightforward too because we can just...".

You can have an audit log without ES. You're also going to hate versioning events. And navigating your tangled nest of events that you accidentally accumulated in the name of agile isn't the nirvana you were hoping for.

I believe it's better to have a system that responds to the requirements of here and now, leaving history behind, learning from mistakes going forward and accepting that yesterday is done.

In my experience, in growing software systems, you're more likely to have not to have even had the data/event at all, than to not have misappreciated what you already had. So event logs feel more like a bunch of baggage ready to trip teams up than a wealth of value.

Re:The Reactive Monolith – How to Move from CRUD to Event So... by the_duke on Wednesday September 29, 2021 @11:42PM (#28702951)
Event sourcing could be a great architecture for general purpose applications - if we had good frameworks and specialized event stores built for this purpose.

Some features that would be needed:

* Automatic , schema based event and aggregate versioning

* A combined event and aggregate data store with ACID guarantees, schema validation and indexes

* event migrations (so you don't have to carry around old event versions forever)

* Event stream migrations that allow re-writing streams for purging data, gdpr cleanup, etc

* Copy on write based stream clones that can be used for testing with production data

As it stands, such a solution is nowhere to be seen. So you have to cobble together a working system with lots of custom work , which often ends in a complete mess.

Re:The Reactive Monolith – How to Move from CRUD to Event So... by jordanbeiber on Thursday September 30, 2021 @02:28AM (#28703737)
Look at Zeebe[1] for example.

If you declare the business process you want to support - either in code or in BPMN, you're automatically mapping out the events that are relevant to the business.

Having a system that takes care of executing this, and tying the pieces of this workflow together is one example where ES can shine IMHO.

[1]https://docs.camunda.io/docs/product-manuals/zeebe/technical...

Re:The Reactive Monolith – How to Move from CRUD to Event So... by dmitryminkovsky on Wednesday September 29, 2021 @11:47PM (#28702975)
I second this wholeheartedly because this happened to me.

I'm the kind of person who learns by my making my own mistakes (as opposed to learning from others', which is what smart people do)—and boy, did I learn a lot from the mistake of trying to build a general purpose application with event sourcing.

Theoretically, event sourcing is the way all applications should be built. I was first taken by this methodology by Martin Kleppmann's talk "Turning the Database Inside Out" (2014)[0]. Looking back, I think I had always subconsciously felt like there was something wrong with mutating state in place. You lose information. You lose accountability. After all, "accountants don't use erasers," and Martin's talk validated many of my obsessive-compulsive insecurities about data. As Charity Majors points out on Twitter [1]:

> ...while storing raw events may be "expensive" compared to any single optimization, all optimizations can be derived from raw events. The reverse is impossible.

And that's true. The table-stream duality [2][3] means that if you store events, you can later project those events into the exact perfect table for any use-case. But if you transform those events into a projection and store that projection, well, whatever table you store is data you have. You wind up having just an aggregate.

Unfortunately, the technical costs you incur when you attempt to store events instead of tabular data are tremendous. The problems and questions that arise are basically endless. Tons of things you wouldn't even think about with a relational database become complicated design decisions [4]. If there were tooling and frameworks that made this easy, it would be one thing. But, there are not!

Kafka Streams is very cool. If you already have a running business with well-developed models and a large team, event sourcing with Kafka Streams could possibly go quite well. But if you need to get an MVP off the ground, it's just not what you need. What you need is PostgreSQL, perhaps with something like Hasura or postgraphile in front of it. What you need is a lightning-fast way to iterate and flesh out your idea. If that's you, and you're reading this, and you're smart: learn from me—event sourcing: don't do it!

[0] https://www.youtube.com/watch?v=fU9hR3kiOK0

[1] https://twitter.com/mipsytipsy/status/1115537408705957888

[2] https://docs.confluent.io/platform/current/streams/concepts....

[3] https://news.ycombinator.com/item?id=23207123

[4] https://softwareengineering.stackexchange.com/questions/3801...

Re:The Reactive Monolith – How to Move from CRUD to Event So... by jordanbeiber on Thursday September 30, 2021 @01:43AM (#28703557)
I believe ES has it’s uses in discreet systems doing their thing for you.

For me ES makes perfect sense in systems handing a workflow for you. Products like Zeebe[1] and Ubers Temporal[2] comes to mind.

These systems basically becomes your business transaction log and as such ES is just a great fit. (I have no idea whether ES is used in temporal though… never looked at the code)

[1] https://github.com/camunda-cloud/zeebe

[2] https://temporal.io/

Re:The Reactive Monolith – How to Move from CRUD to Event So... by swyx on Thursday September 30, 2021 @02:56AM (#28703867)
hi! I work at Temporal. ftr our founders left Uber 2 years ago and Temporal is an independent startup (that is very much hiring) now.

Short answer is yes, we do, but it's abstracted away so you get the benefits (complete observability, retry/resume from failure) without the downsides (handwriting event sourcing logic and storage).

Long answer... it depends what your definition of event sourcing is. We shared some of our internal debate here: https://news.ycombinator.com/item?id=28149159

Event sourcing is an impt implementation detail for us, but it is not something we push onto our users. Still they benefit from it anyway. very much agree with your statement. happy to take followup qtns :)

Re:The Reactive Monolith – How to Move from CRUD to Event So... by jareklupinski on Thursday September 30, 2021 @01:59AM (#28703627)
> learning from others', which is what smart people do

my grandfather had the same words :)

> What you need is PostgreSQL, perhaps with something like Hasura or postgraphile in front of it. What you need is a lightning-fast way to iterate and flesh out your idea.

optimizing my team's incoming stream stack atm, event sourcing seems so enticing on paper...

Re:The Reactive Monolith – How to Move from CRUD to Event So... by ryandvm on Friday October 01, 2021 @10:01AM (#28718532)
> any article that references Fowler should be looked at with just a little bit of suspicion

Indeed. I don't doubt he's a very clever guy, but all of my most miserable gigs involved phalanxes of architecture astronauts endlessly bike-shedding every design session with the platitudes of Fowler et al.

Re:The Reactive Monolith – How to Move from CRUD to Event So... by copperx on Thursday September 30, 2021 @12:50AM (#28703297)
> any article that references Fowler should be looked at with just a little bit of suspicion. Not hating the guy but he's blown quite a bit of hot air in his time.

I've just started reading some Fowler stuff. What hot air are you referring to? I would like to keep that in mind.

Re:The Reactive Monolith – How to Move from CRUD to Event So... by blowski on Thursday September 30, 2021 @01:26AM (#28703474)
People over-apply his patterns. Some patterns that are useful in niche edge cases suddenly become a ‘best practice standard’, sprayed anywhere and everywhere. My rule of thumb is that you should only use his patterns if you could afford to hire him to code it for you.
Re:The Reactive Monolith – How to Move from CRUD to Event So... by tunesmith on Thursday September 30, 2021 @01:36AM (#28703521)
One of those books has a rule buried in the middle somewhere that I was was in 60-point text on the first page of chapter one. I've taken to calling it Rule Zero: Don't implement a pattern unless it actually solves a problem.
Re:The Reactive Monolith – How to Move from CRUD to Event So... by blowski on Thursday September 30, 2021 @02:00AM (#28703632)
Exactly. The problem is not Martin Fowler himself, but those who pick up his books and ignore those warnings.
Re:The Reactive Monolith – How to Move from CRUD to Event So... by nuclearnice1 on Wednesday September 29, 2021 @10:49PM (#28702638)
What makes event sourcing poor for general purpose stuff?
Re:The Reactive Monolith – How to Move from CRUD to Event So... by mr_tristan on Wednesday September 29, 2021 @11:13PM (#28702797)
This seemed like a great rundown on why event sourcing is complicated: https://medium.com/@hugo.oliveira.rocha/what-they-dont-tell-...
Re:The Reactive Monolith – How to Move from CRUD to Event So... by Spartan-S63 on Wednesday September 29, 2021 @10:56PM (#28702690)
Not OP, but the number of variants in the events makes the cardinality of the state of the system extremely large. Probably too large to be able to reason about fully. It makes weird bugs in the event schema likely impossible to debug should they occur.
Re:The Reactive Monolith – How to Move from CRUD to Event So... by 88913527 on Wednesday September 29, 2021 @11:01PM (#28702713)
This is exactly why 2-way data binding fell out of favor in modern UI development. Describing the problem as "the cardinality of the state of the system extremely large" is a succinct way of putting it. The non-linear increase in possible state quickly makes testing an uphill battle. And it's over when you can't have a comprehensive testing strategy, forget about it.
Re:The Reactive Monolith – How to Move from CRUD to Event So... by Spivak on Wednesday September 29, 2021 @11:43PM (#28702957)
Eh, two-way binding is still alive and well it’s just implemented explicitly on top of one-way bindings. It seems most developers find this easier to reason about but the relationship is still there.

Like if all you’re doing is using the event to take the value from the UI and shove it into the model with setState and re-render then you made a two-way binding.

Wouldn’t it be helpful if you could tell React to just do this for you and save you the boilerplate? Boom you invented ng-model.

Re:The Reactive Monolith – How to Move from CRUD to Event So... by square_usual on Thursday September 30, 2021 @12:10AM (#28703090)
2-way bindings are still alive and well in the big "next gen" web frameworks, Vue and Svelte.
Re:The Reactive Monolith – How to Move from CRUD to Event So... by btown on Wednesday September 29, 2021 @11:36PM (#28702925)
Can this be mitigated by ensuring that historical/legacy events are all transformed to more modern equivalents? Similar in theory to https://stripe.com/blog/api-versioning ? Then you at least have a chain of testable transformations and a reduced cardinality for the actual interactions that reflects the cardinality of all the ways your business users actually use your product today.

Of course, this can be lossy, but is it any more lossy than the non-event-sourced CRUD system which is essentially transforming historical events into "overwrite these fields" events that can only be used for their original purpose?

To me, the reason not to use event sourcing is more that the tooling and developer experience is just really poor - and so decisions like the above, while valuable to think through directly, rapidly become overwhelming. Visualizations of system state are only as good as the code you bring yourself. I'm hopeful that people building tools on top of things like https://materialize.com will start to change this though!

Re:The Reactive Monolith – How to Move from CRUD to Event So... by wrwatson on Thursday September 30, 2021 @12:09AM (#28703088)
The first rule of data modelling is that you should model only what you care about. A good model allows you to store and retrieve what you need to meet your requirements.

Maybe you care about all the events that have ever happened on your incredible journey to reach a list of the customers you have today.

Or maybe it would be sufficient to have been mutating a customers table all along. If that is sufficient, then it would be faster and more straightforward to have done so.

Re:The Reactive Monolith – How to Move from CRUD to Event So... by mistahenry on Thursday September 30, 2021 @03:03AM (#28703893)
This is why I always like to have a `status_history` or `audit` table. You don't have to have event sourcing to still want a trail of who changed what in the system when.
Re:The Reactive Monolith – How to Move from CRUD to Event So... by _3u10 on Wednesday September 29, 2021 @11:03PM (#28702734)
Event sourced systems are actually great, as long as they aren’t branded as such. Most of the time it’s totally stupid and the state of the system isn’t reproducible anyway unless the event code is never modified.
Re:The Reactive Monolith – How to Move from CRUD to Event So... by holoduke on Thursday September 30, 2021 @12:45AM (#28703281)
Extra layer of complexity. More development time. More tech dependencies. Don't use it when not needed.
Re:The Reactive Monolith – How to Move from CRUD to Event So... by bsder on Wednesday September 29, 2021 @11:36PM (#28702929)
Event sourcing is decent when you ingest a lot of things that are mostly one way and immutable once they come into your system.

If you have bidirectional communication, you wind up contorting yourself into a pretzel to bury "state" into your event streams.