Re:From Node to Ruby on Rails
by exikyut on Saturday December 18, 2021 @07:11AM
(#29603895)
I have no delusions I'd be able to viably make a dent in the area (at least anytime soon), but I do wonder how this would actually work.
The optimal solution would of course be to shard both the compute I/O and the storage footprint, so each worker only needed to hold onto maybe 1-100MB of data.
Perhaps some existing (simpler) designs could be modified to "hyper-shard" the compute angle, but would still likely require carrying around a large percentage of the database.
In any case, you'd need an internal signalling fabric capable of (cost-effectively) handling very bursty many-to-many I/O across thousands of endpoints to make consensus work in realtime.
It would honestly be really interesting to see how something like this would work in practice.
The optimal solution would of course be to shard both the compute I/O and the storage footprint, so each worker only needed to hold onto maybe 1-100MB of data.
Perhaps some existing (simpler) designs could be modified to "hyper-shard" the compute angle, but would still likely require carrying around a large percentage of the database.
In any case, you'd need an internal signalling fabric capable of (cost-effectively) handling very bursty many-to-many I/O across thousands of endpoints to make consensus work in realtime.
It would honestly be really interesting to see how something like this would work in practice.