News for hackers. Stuff that matters.
Re:From Node to Ruby on Rails by sam0x17 on Friday December 17, 2021 @01:09PM (#29595880)
> Huh. With the context of the rest of the comment, I realize (the very obvious comparison) that a database engine designed to shard to many thousands of small workers could potentially be a very attractive future development path.

Yes. I've been patiently waiting for the database community to realize this for the last 5 years now :)

Re:From Node to Ruby on Rails by exikyut on Saturday December 18, 2021 @07:11AM (#29603895)
I have no delusions I'd be able to viably make a dent in the area (at least anytime soon), but I do wonder how this would actually work.

The optimal solution would of course be to shard both the compute I/O and the storage footprint, so each worker only needed to hold onto maybe 1-100MB of data.

Perhaps some existing (simpler) designs could be modified to "hyper-shard" the compute angle, but would still likely require carrying around a large percentage of the database.

In any case, you'd need an internal signalling fabric capable of (cost-effectively) handling very bursty many-to-many I/O across thousands of endpoints to make consensus work in realtime.

It would honestly be really interesting to see how something like this would work in practice.