News for hackers. Stuff that matters.
Re:Everything wrong with databases and why their complexity ... by morsecodist on Tuesday January 09, 2024 @02:08PM (#38930479)
This is more an advertisement for a type of database than a statement that they are unnecessary.

From what I can tell in the article it seems their differentiator is Event Sourcing and having arbitrary complex index builders on top of the events. It seems similar to EventStoreDB[1].

I have always been interested by the concept of an event sourcing database with projections and I want to build one eventually so it is interesting to see how they have approached the problem.

Also they mention on their site:

> Rama is programmed entirely with a Java API – no custom languages or DSLs.

It makes sense why they have gone this route if they want a "Turing-complete dataflow API" but this can be a major barrier to adoption. This is a big challenge with implementing these databases in my opinion because you want to allow for any logic to build out your indexes/projections/views but then you are stuck between a new complicated DSL or using a particular language.

1: https://developers.eventstore.com/server/v23.10/#getting-sta...

Re:Everything wrong with databases and why their complexity ... by pdonis on Tuesday January 09, 2024 @03:01PM (#38931204)
> This is more an advertisement for a type of database than a statement that they are unnecessary.

That was my reaction as well. The article's claimed argument against databases is that they are global mutable state, which is supposed to be bad. But none of what the article advocates for gets rid of global mutable state (which of course you can't do in any real world application, because the real world itself is global mutable state and anything that deals with the real world has to deal with that). It's just hawking a particular way of storing global mutable state.

Re:Everything wrong with databases and why their complexity ... by a-saleh on Wednesday January 10, 2024 @02:36AM (#38937322)
T.b.h. when looking at things, global mutable state might not be bad, if it is principled?

I.e. I did like how Clojure had it's atomically updating refs.

And you realisticaly won't get more principled than with traditional RDBS, with it's ability to transactions, rollbacks, e.t.c.

Only alternative that I have seen proposed to using DBAS while not relaxing the guaranties, is to double down on event-sourcing and not having mutable-state anymore. Everything could be just a fold over all of the incomming events, right? But I don't think I have seen non-toy example of that anywhere I worked at.

Re:Everything wrong with databases and why their complexity ... by pdonis on Wednesday January 10, 2024 @11:14AM (#38940180)
> double down on event-sourcing and not having mutable-state anymore.

Event sourcing doesn't get rid of mutable state. The event store gets mutated every time a new event comes in.

Re:Everything wrong with databases and why their complexity ... by jdminhbg on Wednesday January 10, 2024 @01:31PM (#38941606)
That's not really what people mean by immutable though. What they mean is something halfway between the casual definitions of "immutable" and "persistent," in that your copy of the data at point X will always be valid, even if new data comes in later.
Re:Everything wrong with databases and why their complexity ... by pdonis on Wednesday January 10, 2024 @02:47PM (#38942296)
> That's not really what people mean by immutable though.

As far as the term "global mutable state" is concerned, which is what the article claims is bad, the event store is global mutable state. One can think of the individual event records in the store as immutable (they don't change once they're recorded), but the article is talking about the entire store, not an individual record in it.

> your copy of the data at point X will always be valid

If your copy of the event store is out of date (missing some events), it's not valid. A new event that your copy hasn't yet registered might change the results of queries on the store, meaning that your copy will be delivering out of date, hence invalid, query results. You can't handwave around that by gerrymandering the definition of "immutable". The simple fact is, as I said, that the real world is global mutable state, and if your app is dealing with the real world, it has to deal with that.

Re:Everything wrong with databases and why their complexity ... by a-saleh on Thursday January 11, 2024 @07:43AM (#38951471)
Append-only log-based data-stores are close enough in my book.

And conceptually, you can then push all of those "what if I am missing events, or tey are in wrong order" into the fold over the data.

Because real world is mutable data, but history of the world at any particular point isn't. It can be incomplete, it can be a personal view, I like that there are things made explicit that otherwise wouldn't. Nice abstraction.

Like, if you are splitting hairs, is even array-append in Haskell immutable? It does create new data somewhere in the memory. But I don't think that makes Okasaki's thesis and book any less valuable :)

Re:Everything wrong with databases and why their complexity ... by pdonis on Thursday January 11, 2024 @11:48AM (#38954758)
> Append-only log-based data-stores are close enough in my book.

Close enough to what? I get that there are advantages to storing global mutable state this way. I just don't see the point of denying that what you are doing is storing global mutable state--every time you append a new log entry to the data store, you mutate the state.

> history of the world at any particular point isn't

Sure, I get that too, and I get that it's nice to have that stored explicitly instead of having to reconstruct it for the particular cases where you need it.

But if someone is running a query on your database, except in some particular cases, they don't want the history of the world at some particular point to dictate the response. They want the most up to date history of the world to dictate the response. And that changes every time you append a new log entry to the data store. Again, I don't see the point of denying that that's what is going on.

> is even array-append in Haskell immutable?

The point of "immutable" in Haskell is that every object is immutable. So array-append in Haskell does not take an existing array object and append an element to it (as the corresponding operation would for, say, a list in Python). It constructs a new array object that is the same as the old object with the new element appended. So the old array object is not mutated.

But the array-append operation of course does change the global state of the program, as you say. So there is no such thing as an immutable global state of a Haskell program. And of course the same would be true for a program in any language.

> I don't think that makes Okasaki's thesis and book any less valuable

Of course it doesn't. But it also doesn't mean the definition of "immutable" that he is using is the only possible or useful one. The article under discussion here is using a different definition, which for the topic it is discussing is more useful. Once more, I don't see the point of trying to twist words around to deny that.

Re:Everything wrong with databases and why their complexity ... by a-saleh on Thursday January 11, 2024 @02:41PM (#38957768)
The article calls out event-sourcing and materialized views as a solution.

I reread the article twice now, and I don't see a conflicting definition, that would contradict i.e. append-only-log being immutable.

Unless your argument is that the author is in-denial about benefits of immutability, because it can't be achieved in this weirdly strict form you seem to propose?

But maybe I need to reread second paragraph of the article one more time?

Re:Everything wrong with databases and why their complexity ... by pdonis on Thursday January 11, 2024 @03:14PM (#38958320)
> The article calls out event-sourcing and materialized views as a solution.

Yes, I know, and it may well be a good solution to various issues with conventional databases. I just don't see how it's a solution to global mutable state, since there still is global mutable state.

> I don't see a conflicting definition

The article never gives an explicit definition of "global mutable state". It also does not explicitly say that its proposed solution gets rid of global mutable state. The section "Global mutable state is harmful" strongly implies that its proposed solution will somehow get rid of global mutable state, but the article never actually says that it does. It just goes into a bunch of other benefits of event sourcing and materialized views, all of which might well be valid benefits, but none of which amounts to getting rid of global mutable state.

> Unless your argument is that the author is in-denial about benefits of immutability

I can't see into the author's mind, so I don't know if the aspects of the article that I described above were inadvertent, because the author simply hasn't really thought through what "global mutable state" actually means, or whether it's on purpose, to imply a claim about the article's proposed solution that sounds good but isn't actually true, as a marketing tactic.

Re:Everything wrong with databases and why their complexity ... by pdonis on Thursday January 11, 2024 @05:57PM (#38960691)
> I don't see a conflicting definition

As I said in my earlier response, the article doesn't explicitly define what "global mutable state" is, but it does say that conventional databases are global mutable state the same way that global variables in an ordinary program are.

By that definition, an append-only log as a "single point of truth" data store, which is the article's proposed "solution", is also global mutable state. The article does not acknowledge this, but it seems obvious to me.

Re:Everything wrong with databases and why their complexity ... by adammarples on Wednesday January 10, 2024 @02:01PM (#38941875)
No, they store data in 'depots', which are append only logs, like kafka. There is no mutable state.
Re:Everything wrong with databases and why their complexity ... by pdonis on Wednesday January 10, 2024 @02:48PM (#38942301)
> There is no mutable state.

Yes, there is. Appending a new log entry to the depot mutates the depot. It doesn't mutate earlier records in the depot, but the depot is the entire record set, not just individual records. The record set is different after the append than before it.

Re:Everything wrong with databases and why their complexity ... by nathanmarz on Tuesday January 09, 2024 @02:34PM (#38930795)
True, Rama does have a learning curve. It's easier to explain why its indexing capabilities are so powerful, as data structures vs. data models is fairly relatable to most devs. But I actually consider its dataflow API to be the bigger breakthrough. Our Twitter-scale Mastodon implementation and examples in rama-demo-gallery are demonstrations of this.

Also, Rama has a first-class Clojure API (I should probably update the website).

https://github.com/redplanetlabs/twitter-scale-mastodon https://github.com/redplanetlabs/rama-demo-gallery

Re:Everything wrong with databases and why their complexity ... by morsecodist on Tuesday January 09, 2024 @03:16PM (#38931466)
Cool on the clojure API. Functional programing really fits with this sort of thing. Does your implementation limit you to the JVM?
Re:Everything wrong with databases and why their complexity ... by nathanmarz on Tuesday January 09, 2024 @03:55PM (#38931983)
Yes, it's JVM-only for the foreseeable future.
Re:Everything wrong with databases and why their complexity ... by worik on Wednesday January 10, 2024 @01:44PM (#38941727)
That is sad

Why take that path?

I get your point about global mutable state, but I do not see how you have done anything more than shuffle the complexity around

Being dependant on software like the JVM makes me very reluctant to investigate further. The JVM is not involved in anything I use, I do not think i want it.

Re:Everything wrong with databases and why their complexity ... by zubairq on Wednesday January 10, 2024 @02:31AM (#38937286)
Agreed that this article is an advert for a certain database type, but the advert for red planet labs worked. I certainly read it! :)
Re:Everything wrong with databases and why their complexity ... by andreareina on Wednesday January 10, 2024 @02:40AM (#38937347)
It also turned me off from them so there's that.
Re:Everything wrong with databases and why their complexity ... by zubairq on Wednesday January 10, 2024 @02:42AM (#38937367)
Good point. I don’t use red planet labs product myself either, but in a world where it is hard to get people to even consider your product the article did work
Re:Everything wrong with databases and why their complexity ... by wavemode on Wednesday January 10, 2024 @01:39PM (#38941687)
Sure but, were you seriously considering them beforehand?
Re:Everything wrong with databases and why their complexity ... by mike_hearn on Wednesday January 10, 2024 @03:52AM (#38937891)
A Java API doesn't mean you can only access it from Java. Such APIs can also be accessed from most languages. Obviously bytecode languages like Clojure, Kotlin, Scala, Groovy etc. But also via Truffle languages like JS, Python, Ruby. All without bindings.

You can also do C++ and Rust, but that does require a bit of binding work.

So it's not really limiting you to a single language, or these days, even a single runtime.

Re:Everything wrong with databases and why their complexity ... by vidarh on Wednesday January 10, 2024 @04:44AM (#38938291)
Assuming you're willing to run a JVM and possibly specific language implementations. That's fine for a lot of people, I'm sure, but it's not something I'd be willing to do for what is fairly little benefit over "just" whatever solution you're already familiar with to use as the depot and the "P-stores" with workers in between.
Re:Everything wrong with databases and why their complexity ... by vidarh on Wednesday January 10, 2024 @04:41AM (#38938269)
Yeah, the Java dependency makes me reject it instantly - the only advantage they potentially could have over "just" using any database with ability to query remote databases is not having to roll your own, but the barrier to rolling my own is lower for me than introducing a Java dependency.

(A replicated set of databases for the depots; a set of workers creating the "p-stores" in ... another set of databases, and you've replicated the architecture; heck you can even run this entirely off postgres w/replication + triggers to handle partitioning and materialization with no external workers if you like)

EDIT: Also their API fairly unambiguously is effectively a DSL, just one without it's own separate syntax and parser allowing access from outside the JVM.

Re:Everything wrong with databases and why their complexity ... by lambda_garden on Wednesday January 10, 2024 @05:13AM (#38938490)
> the barrier to rolling my own is lower for me than introducing a Java dependency.

Really? I never had problems using some JVM base image and deploying via a Docker image, which is what I would be doing anyway.

Re:Everything wrong with databases and why their complexity ... by vidarh on Wednesday January 10, 2024 @05:18AM (#38938533)
I've had enough problems with JVM deployments over the years to just not want the hassle in order to use something which adds marginal value. If you don't, then by all means, use it - I'm not suggesting it's bad for everyone. If it provided some massive major advantage over the large number of other options, I'd consider it too, but I don't see anything that indicates that's the case here.
Re:Everything wrong with databases and why their complexity ... by adra on Wednesday January 10, 2024 @11:07AM (#38940078)
Do you reject Kafka because it runs on the JVM? This is a ludicrous position to take. You may not want to fund development of projects when you're forced to train devs to work them, but to reject using technologies based on the programming language / framework is just fundamentally flawed. Don't use Python tools because I hate python. Don't use kubernetes because it's golang don't use AWS because it has a bunch of java services. S3? I'm not entirely sure but I wouldn't be surprised if it was java edge service based as well.
Re:Everything wrong with databases and why their complexity ... by vidarh on Wednesday January 10, 2024 @11:54AM (#38940607)
Note this line:

> If it provided some massive major advantage over the large number of other options, I'd consider it

People reject using technologies because of the impact of having to support them all the time when the benefit does not justify needing to support one more technology. I don't have an issue with using JVM based services if I can have someone else manage them for me, but when I have to deal with the operational issues myself, or within my team, then how comfortable we are with managing a specific technology vs. the advantages it brings over competing alternatives absolutely is part of the consideration, and it'd be irresponsible for it not to be.

In this case the benefit seems marginal at best, and I'm not deploying a JVM based service for marginal benefits given how much hassle they've brought me over the years.

Re:Everything wrong with databases and why their complexity ... by isbvhodnvemrwvn on Wednesday January 10, 2024 @11:16AM (#38940201)
I need to use Java exactly zero times when interacting with those, they have dedicated clients for everything, something this stuff apparently is proud of not having.
Re:Everything wrong with databases and why their complexity ... by randomdata on Wednesday January 10, 2024 @12:48PM (#38941120)
> Do you reject Kafka because it runs on the JVM?

Nothing is outright rejected because it runs on the JVM, but it is a detracting quality, adding complexity. If Kafka is clearly the best solution available for your problem, you'll suck it up and the use JVM, but if there is another approach that will also serve your needs without a JVM dependence then the choice is easy.

Re:Everything wrong with databases and why their complexity ... by worik on Wednesday January 10, 2024 @01:48PM (#38941767)
So now you are dependant on JVM and Docker
Re:Everything wrong with databases and why their complexity ... by cryptonector on Wednesday January 10, 2024 @11:53AM (#38940604)
The API could be LinkQ-like, so it'd still be SQL-ish, just in a query builder sort of way.
Re:Everything wrong with databases and why their complexity ... by vintermann on Wednesday January 10, 2024 @12:15PM (#38940801)
Well, it is an advertisement, but much of what it says is undoubtedly true. It's true that databases are effectively global mutable state in practice, with all the headaches that gives, and that event sourcing and materialized views is a damn appealing solution to it.

However, the jump from that to saying you have it is a bit like the jump from talking about the messiah and saying that he's sitting in the next room just now.

We investigated event sourcing and materialized views as the backbone of a (new) big business application five years ago. The problem was that there were so few products back then and they were so untested (unless my mind is playing tricks on me, we did look at EventStoreDB, as well as Kafka streams), and despite being open source they were also usually heavily tied to one commercial provider. No one had anything like experience with it. I couldn't convince my colleagues to go for this, much less my bosses. Hell, I could barely convince myself.

Philosophically, I believe it's sound, but how much has changed in those years?

Re:Everything wrong with databases and why their complexity ... by foofie on Wednesday January 10, 2024 @01:15PM (#38941428)
> This is more an advertisement for a type of database than a statement that they are unnecessary.

I stopped reading the article once it was evident that the author was not making any case regarding databases being unnecessary, and instead was putting up a series of strawmen that sounded like desperate attempts to reject all alternatives to the thing they were selling.

The author's attempt to talk about data structures was particularly baffling, knowing that pretty standard RDBMS such as PostgreSQL already support JSON data types. Are we supposed to forget about that?