elendilm
I find it unlikely, especially due to the numerous pathetic suggestions LLMs keep offering me on the architectural discussions I engage in.
Verbal abuse is what makes it useful to me. A punching bag where it infuriates me making me lay down the how and why its suggestions are stupid. This results in progress.
I guess we would have to wait and see. Maybe LLMs do get better at architecture in the future. But seems unlikely to me without expert supervision.
>> Writing such a complex software as database engine requires strategy, awareness and rigor.
> Correct. Current LLMs are pathetic at architectural correctness without expert supervision at every turn.
> Try having a discussion on designing an architecture for a new kv engine with cache-oblivious algorithms that beats lmdb on generic random point-key lookups on a disk-backed mmap file.
> What happens in the future is up for debate,
Unable to see the architectural problem mentioned is just icing on the cake.
How do you do this?
darkwi11ow, in the sibling comment, gave a substantive response on the underlying mechanics with a link. Godel's incompleteness theorems also yield similar insight. But I want to shine some light on a different phenomenon.
What happens in the future is up for debate, but the current architectural reasoning capabilities of LLMs are pathetic.
Try having a discussion on designing an architecture for a new kv engine with cache-oblivious algorithms that beats lmdb on generic random point-key lookups on a disk-backed mmap file.
Or try any novel architectural discussion and see for yourselves.
Modern LLMs cannot even confidently state many true sentences as "maximally" true as possible without hedging and tone policing as if we were doing politics ;).
Hence an expert is required at every turn.
Yes. The combinatorial explosion is a great assessment of the "why".
But the problem is more nuanced than that because one has to assume that LLMs can reason accurately all the time which is demonstrably false.
The current LLMs cannot even state true sentences as true if they sound confident, without hedging.
Future LLMs may mitigate this, though it looks unlikely.
Making it distributed opens a whole can of worms, which is where Jepsen shines.
Seasoned developers are unimpressed, or even disgusted, by the same massive piles of code.
But in recent years, the phenomenon was visible with Dip which was built on rust.
Dip does massive parallel fanout using green threads ('may' coroutines) with work being NUMA pinned (software hash) using a thread per core model.
A default request may span up to _chunk_count_ jobs (default 5k) configurable per request.
Every request (x 5k jobs) from the client on a stress test sees a jump of ~10-20 MB of memory. Sometimes the jumps use large swaths of memory - say 100s of MB and stabilizes for a while only to gradually climb up again later. Rinse and repeat - OOM kicks in and the process crashes.
Many times we thought we were leaking memory somewhere. But armed with the Qt experience a decade ago, we intuited maybe its the allocator - and switched to mimalloc on desktop executable.
Dip's memory, using mimalloc, stabilizes at ~60 MB post request completion no matter how many requests you throw at it.
Pity that on mobiles, more specifically Android, we have to live with the default allocator on even trivial Qt apps let alone embedded Dip. But people have lived with it for at least a decade, so it is what it is.
Thus on mobiles, we just have to be content like everybody else, unable to squeeze every last ounce of performance and efficiency out of the hardware on stress tests ensuring Dip remains "just better :(" :)
Eg: < Optimize for the slowest resources first (network, disk, memory, CPU) in that order, after compensating for the frequency of usage, because faster resources may be used many times more. For example, a memory cache miss may be as expensive as a disk fsync, if it happens many times more.>
This is just obvious common sense. Great read.
This is for a long running process where consuming compute and memory should only occur on processing a request and must revert to a baseline memory consumption once the request is completed.
malloc with default allocator always gradually increases the memory consumption. This continues with enough request bursts on our testing until the process runs out of memory. mimalloc doesn't.
On mobile, we are stuck with the default allocator. On mobiles, the phenomenon of the default allocator consuming memory is reproducible even with normal Qt apps. It is what it is.
The logic stands. A reviewer, even those who get zero compensation, can always review publicly published papers just like others.
It's his/her prerogative to keep an online footprint or not.
It's fairly basic. For a work published on a public platform, the same guys paid by the journal can also review it if they so choose to in addition to those people in the public who were not paid by anyone.
We have smart batching in Dip where we deduplicate work received in what we call a "batched" request. We use BTreeMap to deduplicate work in the preprocessing "scatter" phase.
This deduplication in smart batching is the slowest phase on most bulk write workloads (200k+ inserts in a single request) and takes sometimes up to 3x the time taken for the combined fanout + db writes + db commits + gather.
Will try out with Bbtree and see how it goes.
Many don't want to say it or admit it.
True. This is true by experience. I want to add that, additionally this is not just true based on experience, rather it is fairly a straightforward logic.
If the cpu is already saturated at 100% doing work in the most "efficient" way possible for a given data structure (some sort of a shannon limit for the given data structure), then there is zero further improvement possible.
Any additional improvement must therefore happen either on the thing that the CPUs operate on (data structure improvements) or on the hardware themselves.
When the Uber blog link was provided with exact scenario in the excerpt, you pivoted to how their new version has schema enforcement. You claimed you were able to prove all inefficiencies you encountered were due to bad data modeling or bad decisions which went against RDBMS.
When pointed out that the newer blog post doesn't mention removing the secondary indexes which was the whole issue, you digressed to the grand finale of absurdly claiming Sqlite will saturate phone's I/O before one saturates Sqlite, which is so false that it is not even funny. And finally demanding our Dip's benchmarks (which will come in due time, if? we ever release it to the public).
All of this when more than enough hand-holding was made to help you understand. Even when the underlying physics was easy to understand. Enough pivoting.
Let our conversation stand and be of help to anyone curious enough to learn the tradeoffs discussed. Best wishes.
Some are just physics. Say if you want SSD write saturation and multiple parallell write transactions, the traditional RDBMS model itself may not suffice. Its not just data size but concurrent write transactions, lock contention, etc. Additionally, think resource constrained devices like mobile phones. Or an aggressive cost reduction mentality where each cpu cycle shouldn't be unnecessarily wasted without feeling guilty. If you come from a systems engineering background like that, you wouldn't be much impressed with many traditional databases at all. I am generally not much interested in SQL databases anyway.
> They needed to create secondary indices as tables because they were storing everything in JSON blobs, which is difficult to directly index
Our database Dip stores raw binary as well as json as first class citizens. Can't speak for Docstore, but as far as I see, it is just an evolved form of its earlier self with different tradeoff according to their workload. Note the lack of mention of the removal of secondary indexes as separate data.
> I would counter that if you keep needing schema changes, you haven't done a good job modeling.
Disagree. Schema stabilizes - yes. But doesn't mean they don't go through cycles of change occasionally.
That is not the case. Even on a single machine with multiple shards (think hundreds or even thousands of kv engine say LMDB envs), a distributed query parallelizes efficiently. Additionally it can also saturate the SSD/NVMe with multiple write transactions concurrently.
It definitely is the first option and not a workaround. Even replication benefits from this write saturation and reduced latency and parallelism. Naturally, multiple writes involving the same logical unit of work can go through a durable execution framework (think ordered-log/kafka) as one insert and the client dismissed only for the consumer later attempting multiple idempotent writes (logical work with retry on failure) - achieving maximum SSD saturation. The client is later informed of the status.
> Most of the big offline use cases I’ve seen are aerospace/drones or logistics or robotics ...
Our previous ERP (built almost over a decade) for a large enterprise client, required disbursing customers as the absolute paramount priority regardless of internet availability - which is where offline first became the only choice. They wanted data to seamlessly move across thousands of offline systems - which is where we wrestled with replication/conflict resolution, etc. Our new Slyp and SlypBusiness is a reincarnation/re-imagining of this old ERP with added goodies but as mobile first applications.
There is only up to a point where we tolerate other databases, and beyond which the conventional wisdom starts to be a net negative and we realized we are better off building one for ourselves.
You seem to be confusing writes/second with concurrent write transactions/second. Nvme won't rescue you if Postgres locks up only allowing each write txn to proceed sequentially when what you need is SSD write saturation on separate write transactions on multiple shards in a single machine.
> You'd be way better off going all NVMe today as your first step to "scaling" (in fact you are insane if you are using anything else for an OLTP database), and just that is likely to immediately solve whatever problem you have already.
This is hilarious. So we should all start building based on your prescription.
> what exactly is the benefit of manually implementing what it already does well?
Really? Is that a serious argument. Postgres is not a magic database that does everything that everybody wants very well.
In systems engineering, the trade offs are obvious. But if it is not clear at this point, I don't think I can offer much help.
I will offer one last advice.
Just write a snippet of code inserting 100M small keys and values into, say 1k chunks/shards of separate kv engine (LMDB, whatever). Do normal hashing for shard selection. Now store secondary indexes likewise. Now do parallel fanout for secondary indexes for selecting some 100k data. Collect the index data and do another parallel fanout and fetch the final data. Now go see the latency results (excluding lmdb env setup time). Additionally observe the latency for writes :-)
Rich index metadata is just icing on the cake. Cheers.
Composed of the right primitives at sufficient granularity, we may yet approach the one true hammer :)
You have demonstrated bad faith beyond any reasonable doubt.
Your concern has already been addressed in my parent comment had you paid attention.
Cheers.
Uber for one as documented here https://www.uber.com/us/en/blog/schemaless-part-one-mysql-da...
There are many more and most may not be documented online.
Our usecase is offline first mobile ERP (roll-out hopefully soon) with online-sync capabilities doing invoice/coupon processing, etc. in a network of regional shops - with a focus on minimizing latency and aggressive reduction of costs at scale.
Once we measured the reduction in latency due to parallel fanout (over index shards - not just data shards) for range scan, there is no going back ever again.
When you need the same database to be running on mobiles and servers alike, you need embedded database for mobile and standalone executable for server.
> they will perform better than the ones built with coordination with the storage engine and everything else?
No amount of coordination with the storage engine can override the physics of lock contention in distributed systems.
> This seems a very unusual set of circumstances so I'm very curious on the use case.
Not with my background. Its the norm :)
Not just data, but concurrent operations per second for writes and reads involving seperate transactions. Eg: Scaling concurrent write transactions on different shards with minimal write lock contention.
> Who actually needs this scale?
Many do including Uber https://www.uber.com/us/en/blog/schemaless-part-one-mysql-da...
> You can just do this with postgres.
You seem to have missed the final sentence beginning with "Note:" in my original comment. Nobody said you can't do this with postgres.
> Most people don't even get that right. And they're going to suddenly get it right and implement efficient indexes?
What most people get right is not my concern. If you find building a thin wrapper layer for your application as hard, clearly this may not be applicable to you.
Yet since you asked, assuming good faith, here is what you are looking for.
Following is an excerpt from https://www.uber.com/us/en/blog/schemaless-part-one-mysql-da...
< The indexes are eventually consistent; whenever we write a cell we also update the index entries, but it does not happen in the same transaction. The cells and the index entries typically do not belong to the same shard. So if we were to offer consistent indexes, we would need to introduce 2PC in the writes, which would incur significant overhead. With eventually consistent indexes we avoid the overhead, but Schemaless users may see stale data in the indexes. Most of the time the lag is well below 20ms between cell changes and the corresponding index changes >
Hint: Extreme scale. You anyway have no full ACID guarantees at scale and one has to implement their own relevant guarantees and cache invalidation at extreme scale. You either have to implement your guarantees or pay the cost of bottleneck and latency.
They are achieved via idempotency, durable execution, etc.
There is sufficient reason for me to think that either you may not have read my comments attentively enough or you may lack the experience of building reliable systems at scale.
A developer need not be hand-held with this reasoning on the topic of scaling. I have given ample reasoning to reconstruct.
I advise do your own research.
If the database is running embedded in your application, then there is very little extra cost involved.
If one is using the database as a separate process, you incur additional latency for the network round-trip + the payload of the matched index items.
That is the cost you pay.
But in return you get natural parallel fanout, reduced locks/contention bottleneck, etc.
You also get to have your indexes in dedicated shards on dedicated nvme hardware. With batching, you can reduce the contention on indexes even further.
You also get a simpler mental model, simpler queries, more deterministic and predictable latency. The list goes on.
Application-managed indexes are ideal for extreme scaling.
Why would you think that?
No, not at all. Application maintains indexes as separate normal data/row via CRUD operations into a dedicated table/collection with references to the original instead of letting the database do it on your behalf.