I mean that conclusion is true of basically every complex system. That's the whole point of hacking together an MVP first, to see what really matters.

However it's silly IMO to use that criteria as an example of why not to do event driven architecture. It's well understood in the eda community that you generally don't start with streaming systems unless you know you need it from the start.

In this case it was shiny object syndrome on the part of a few early architects.

The whole project could have been a Rails app.

The lesson is less about event sourcing, and more about picking architectures in light of business needs rather than trying to be fancy.

I'm actually quite happy to see that other people are also facing similar problems with event-sourced microservices. The project that I'm on (currently working for a neo-bank) has been trying to use event-sourcing from the get go, and oh god, is it a mess. The project has been going on for awhile and some of the devs thought it a good idea to focus on scalability and all the other metrics that don't matter.

As your data evolves and your schema changes, you'll have a mix of messages of both the old and new schema in the same topic. You now have to change your consumer services to make sure you can handle the new schema as well as the old schema if they are important (think reconsuming in Kafka). Your code then gets really ugly having to handle OldXXXEvent NewXXXEvent and loads of if-else statements sprinkled in the code base. Either that, or migrate your data to a new topic which is a one-off exercise that takes time which you'll then have to do for different environments.

I'm sure event sourcing has its place but I'm not entirely convinced the approach is necessarily better than just plain ol' database which would have saved us more time and get our product out quicker.

This, this and third time this. I have the exact same experience. You should know the business requirements very well before you choose event sourcing and hope they don't start changing drastically.

Your word of caution is important. For some projects it's not initially clear what the right data model and bounded contexts are. And for some projects it's just overhead. But the converse is also true: for some projects/systems it turns out that it being events first is the _only_ way for it to work. I've encountered that the last few years with systems in logistics that show an integral view across parties.

The way we approached that is that we started with a standard system and refactored to event processing approach when the decomposition into bounded contexts was clear.

So in that sense it's similar to the right way of approaching microservices: start out with a 'monolith' that you split up.

A way to deal with it that I've seen to work with Kafka is to make your messages not last very long (say about a week) and make them explicitly idempotent. So an "old" message being run would not affect the app negatively.

And then you make every producer of messages able to reproduce all the messages it knows about.

So if you have a new service and need historical data, you ask all of your dependencies to resend the data, and existing services should not be affected.

Schema evolution is natural - old messages with old schemas don't last very long, and you can slowly migrate services to new schemas as needed.

If you structure your events that each one holds all state of an entity, you could get removal of data easily for free, as any new message would overwrite old state.

Though the whole thing has its own problems though - especially when you go into large amounts of messages/data. Keeping copies of those around if you make a lot of changes can be quite expensive.

So the whole event sourcing thing looks to me as it needs a decade or two to mature so that tools are built and best practices established. It looks to me as basically a way to keep data for a lot of separate services in a fluid eventually consistent way.

Wonder if Clojure's Datomic is already there. Haven't used it myself but have read promising things about it.

Best practices around this have already been established. Most if not all event stores - which Kafka is not - have a concept called 'position.' You save the position atomically along with whatever you did with the message. Then if you crash, you simply ask for all messages starting from that position. If you have a new service (or a new projection), your position is 0 so you get everything.

That is indeed the case, and this is what Kafka calls unlimited retention topics. This however hampers schema evolution significantly as you're not allowed to make backwards incompatible changes. Or rather if you allow those changes, every service would need to be able to handle every schema throughout time.

If you make the retention to only several days this would mean you can evolve your schema with even breaking changes. Consumers would need to support only schemas that are "active right now". If the retention period is long enough that it is safe to assume all the consumers have acted on all the old messages you can delete those messages. You would need a mechanism to replay them though, if that data is ever needed again, but it would just be in the latest schema.

I feel like people are running into these problems because they want to pretend that a message broker is an event store. I could try to shovel a star schema into MongoDB too, but why would I want to?

Keeping data only in the latest schema is dangerous. We have no idea what data the business will find useful years down the line. By only having whatever is in the latest schema, you may have thrown valuable data away.

Datomic is interesting in the context of event sourcing, because it can be thought of as a highly generalized event store system itself, with each database transaction being an event.

+1.

My company consulted on a project where the use of event sourcing was THE reason the entire project (and the client company) failed.

This was the event sourcing library being used for that https://github.com/johnbywater/eventsourcing

Was the actual library used a significant part of the problem, or was it the pattern of event sourcing itself that wasn't a good fit?

Neither. If Richard's statements were facts they would violate his NDAs. In fact, these statements are falsehoods, and seem designed only to cause damage. Also, he's referring to 2016!

The only "problem" is that in 2016 I was physically assaulted at work, Richard had to leave work soon after, and has been denigrating my work since then. For example, here's a post from 2016, where it appears he had a multi-account conversation with himself: https://news.ycombinator.com/item?id=13129798

Three years later, he's still unable to get over it and move on. This feels like a kind of twisted sexual investment, so I have chosen not to reply.

The Python eventsourcing library is excellent, and event sourcing is a great idea. The library is open source, so if Richard had discovered a bug that caused "corrupt data", he could have raised an issue on GitHub. But there hasn't been such a bug, and no such issue has been raised. Everybody can look through the mere 56 closed issues in the entire history of this successful project to see how many "corrupted data" bugs there have been (zero). https://github.com/johnbywater/eventsourcing/issues

The library is being used successfully in production. Some users have data stores with millions of stored domain events, and the library still works very fast. I didn't hear about anybody having billions of events yet, but I wouldn't expect much difference in performance, so long as the infrastructure has sufficient volume.

In case the wrong impression is created by listening to Richard's rubbish, SQL and event sourcing aren't somehow incompatible. Event sourcing isn't somehow the opposite of "regular databases". The library works very well with SQL databases, through both Django ORM and SQLAlchemy. It also works with NoSQL databases.

If you see posts like this in future, please ignore them, it's just fake news. My understanding is that Richard has diagnosed mental problems. But perhaps there's a kind of nominative determinism from the shortened version of his name? I don't know. At any rate, I've been keeping a log of these occasions, in case I need to call the police again.

Both. The library was slow and corrupted data and they were using regular databases along with the event-sourced stuff. That would never work, the whole system would have needed to be event sourced for it to have any chance of working.

It was essentially a CRUD app with the need for history of changes (for a few tables), SQL was the correct way to do it.

Couldn't agree more. It was disastrous for us with no real benefits.

I'd love to read about the original assumptions -> original design -> revised assumptions -> design invalidations if you care (and are able) to write.

Unfortunately it was on a consulting project with an NDA.

Thank you for this. I've been warning people off Event Sourcing for a while now. The architecture is the most convoluted, pretentious, redundant and downright soul-crushing. If you see Event Sourcing anywhere, run away. Run far, far away.

Some of worlds most useful and powerful data structures are the projection of an event log. The tables of a RDBMS, the balanced writes of a SSD, and even the classic: double-entry book-keeping.

Even the data stream of a TCP connection is a projection of events, which is why (and how) we can reconstruct them by replaying captured segments.

So just because there are some lousy executions of a general architecture, doesn’t mean we should recoil from the basic idea.

My takeaway is that successful event-sourced structures are crafted for the domain they represent. I’ve developed a couple for my own work, for very specific aspects of an application, and they work well in context.

If your experience has been that a general-purpose ES framework leads to shitty, hard-to-maintain apps, I’d say that’s evidence for the corollary.

Just because a tool is powerful doesn’t mean it’s appropriate. There’s a reason most people should just use a DB and not a raw event log. The issue I have with ES proponents is they seem to all pretend there is no additional complexity that comes with it. I think ES is useful but not always and requires weighing the costs and benefits, and we need to be honest about it.

You could say the same about e.g. lock free algorithms. If you work on them you are usually just playing with some cool technology instead of solving the business requirements.

I've only suggested event sourcing once, and that was for keeping physical warehouse inventory synced with orders. I.e. instead of having like a database table with a "available product count", we made a view that calculated "product stock minus pending orders" on the fly at any given timestamp. Very efficient, simple and easy to debug. But as a general system architectural pattern it does seem to create more issues than it solves. Just look at the MailService example in the article. Dude, just use a queue...

Good approach. Much of software engineering is applying principals like event sourcing judiciously. Why does the whole system need to be in one architecture?

I think you're doing it wrong. It shouldn't be any more complicated than writing events as a producer to a Kafka topic.

Gets more complicated when you add in distributed computing, redundancy etc... However those things are always complicated

Simple problem: checking users have the rights to do something before you let them. Do you reconstruct the user account every time you want to check their rights (so, for every action they do)?

No, you should not compute the state you need from the event log on every request, this would be absurd. Your authorization service can maintain its own database (a "view" of the current state), or even an in-memory representation computed at startup, and update it whenever a new event pops up. Alternatively, if you are using Kafka, you can use stuff like KTables to do this.

This sounds like working around event sourcing - what value does event sourcing add here, vs simply not using it at all for user accounts?

For this specific case, nothing, however let's say you need to know when the user got a specific scope and from who. You probably can answer this question in SQL if you prepared your database to do it (i.e: a changeset table), however in an event sourcing architecture you would gain this information for free.

However, just because you're using event sourcing doesn't mean you don't have a database with the current state of your entities.

In Ruby on Rails one can add just one line of code to achieve the same result.

  class User < ActiveRecord::Base
    has_paper_trail
  end

That solution is good enough for most use cases and doesn't require highly skilled architects and developers to implement Event Sourcing properly.

The siblings already said it, but event sourcing requires you (at least for practical purposes) to segregate the read model from the write model using "projections". The good thing is that, whenever you create a new service, you can derive the projections that your service will need from the same source of truth. In this way, you create coupling on the data, but not on the concrete service that owns it.

This is not very different from what a relational database does with redo logs. In fact, in a way, using event sourcing resembles composing a system from the fundamental building blocks of a traditional DBMS, in a distributed way.

That's command-query responsibility segregation. The command stream gives you the nice log of everything that happened, while the read database is easy to query (and to reprogram if necessary).

Isn't that confusing two different things, CQRS and ES? I use CQRS all the time, but have never built a system using Event Sourcing.

Regarding logs, "traditionally", I'd use an audit/change history table for objects that needed one.

I got handed down an Event Sourcing project because the client left the original developer because he kept getting burn-outs.

I can't shake off the project because it keeps getting stuck to the bottom of my shoe.