News for hackers. Stuff that matters.
Re:In-memory vs. disk-based DB: Why do you need a larger tha... by hakunin on Saturday September 02, 2023 @02:52PM (#37364352)
Not 100% what the article is about, just a short story.

In one of my old jobs we had megabytes of infrequently accessed static key-value data. If we simply loaded it into a const (i.e. a hash table), it would blow up the RAM so much, that we would need to upgrade to bigger VPSes. If we put it in database, it would make it annoying to keep these tables up to date, track changes in them.

I figured this was one of those in-between use cases, where the best solution is to have zero-RAM lookups from SSD. In my case, I wrote a little ruby library[1] that arranges data in equal cells in a file, and performs binary searches via `pread`. This was perfect for us, because we kept data in our repos, sacrificed no RAM at runtime, SSD lookups were fast enough, and we didn't have to support a more elaborate db.

[1]: https://github.com/maxim/wordmap

Re:In-memory vs. disk-based DB: Why do you need a larger tha... by rubiquity on Saturday September 02, 2023 @03:16PM (#37364578)
> sacrificed no RAM at runtime

Those lookups eventually made their way into the kernel's page cache.

Re:In-memory vs. disk-based DB: Why do you need a larger tha... by hakunin on Saturday September 02, 2023 @03:31PM (#37364734)
Are you saying that the file would just entirely be loaded into a page cache eventually? I imagine, even if true, it still wouldn't result in OOM killing the server/worker daemons on account of this data?
Re:In-memory vs. disk-based DB: Why do you need a larger tha... by rubiquity on Saturday September 02, 2023 @04:52PM (#37365426)
Depending on the file size and available memory, yes. In the event memory needs to be reclaimed Linux’s memory management system will free from the page cache first if processes need more anonymous memory.
Re:In-memory vs. disk-based DB: Why do you need a larger tha... by coldtea on Saturday September 02, 2023 @07:00PM (#37366216)
The "log(n) * number of searches" page lookups, that the kernel could clean up at any time after a search, instead of all n items that it would have had to load and keep in memory?

Yes, they did.

Re:In-memory vs. disk-based DB: Why do you need a larger tha... by boywitharupee on Saturday September 02, 2023 @03:15PM (#37364573)
Basically, your solution ended up residing on-disk, but the data was fragmented in such a way that efficient lookup was possible? Did this copy from kernel space to userspace, also?
Re:In-memory vs. disk-based DB: Why do you need a larger tha... by hakunin on Saturday September 02, 2023 @03:26PM (#37364682)
>the data was fragmented in such a way that efficient lookup was possible

Yeah, there's a build step that sorts and arranges data into "cells", making binary search possible.

Not sure about kernel/user space. I'm just calling `pread` from ruby, so only a few bytes are loaded per lookup.

Re:In-memory vs. disk-based DB: Why do you need a larger tha... by CraigJPerry on Saturday September 02, 2023 @02:58PM (#37364417)
I think i’m missing something here, could you not just mmap() the file in and let the kernel take care of memory pressure for you?
Re:In-memory vs. disk-based DB: Why do you need a larger tha... by hakunin on Saturday September 02, 2023 @03:03PM (#37364472)
I'm no systems programmer, but I remember trying to research mmap approach. It's been 3 years, so I'm not sure what stopped me, but something didn't feel right. Perhaps it was the lack of control over how memory is used. I could clearly see how not use it, and didn't want any fluctuations.

Edit: oh and I think I did come across some article like the one linked in the neighbor comment. It's starting to come back.

Re:In-memory vs. disk-based DB: Why do you need a larger tha... by sschnei8 on Saturday September 02, 2023 @03:02PM (#37364460)
Re:In-memory vs. disk-based DB: Why do you need a larger tha... by rubiquity on Saturday September 02, 2023 @03:17PM (#37364591)
Good paper and overall the right opinion but not very relevant to a relatively simple system as OP described since they weren't dealing with safety, transactions, etc.
Re:In-memory vs. disk-based DB: Why do you need a larger tha... by legulere on Monday September 04, 2023 @10:40AM (#37380904)
Also not access times, so OP doesn’t care if the OS frees the pages again.
Re:In-memory vs. disk-based DB: Why do you need a larger tha... by lisper on Saturday September 02, 2023 @06:02PM (#37365902)
Sure -- until your system crashes and data is left in an inconsistent state because some of it was written to non-volatile storage and some of it wasn't and your OS had no concept of transactional consistency because mmap is a leaky abstraction.
Re:In-memory vs. disk-based DB: Why do you need a larger tha... by posnet on Saturday September 02, 2023 @06:32PM (#37366084)
Not a problem here. Read the top level comment.

"static key-value data. ... a const (i.e. a hash table)"

Re:In-memory vs. disk-based DB: Why do you need a larger tha... by lisper on Saturday September 02, 2023 @07:01PM (#37366218)
Ah. I stand corrected.
Re:In-memory vs. disk-based DB: Why do you need a larger tha... by legulere on Monday September 04, 2023 @10:41AM (#37380920)
And Flatbuffers for data-layout and access.
Re:In-memory vs. disk-based DB: Why do you need a larger tha... by teaearlgraycold on Saturday September 02, 2023 @04:39PM (#37365318)
Why not commit a sqlite db?
Re:In-memory vs. disk-based DB: Why do you need a larger tha... by hakunin on Saturday September 02, 2023 @05:02PM (#37365498)
I considered that, but couldn't find a way to precisely control amount of RAM used when you read from it.
Re:In-memory vs. disk-based DB: Why do you need a larger tha... by jerrygenser on Saturday September 02, 2023 @06:33PM (#37366091)
I have a similar use case and I have had success reading from sqlite files embedded in my repo