Hmm that’s only 190Hz on average, but we don’t know what kind of search engine it is. For example if he’s doing ML inference for every query, it would make perfect sense to get a few cabinets at a data center. I’ve done so for a much smaller project that only needs 4 GPUs and saved a ton of money.

Nah, it's text-only requests returning JSON arrays of which newspaper article URLs mention which influencer or brand name keyword.

The biggest hardware price point is that you need insane amounts of RAM so that you can mmap the bloom hash for the mapping from word_id to document_ids.

You could have used a sharded database like Mongo. Just throw up 10 shards, use "source" (influencer or brand name) as shard key?

Yes, I could have used Mongo, but it would have been 100x to 1000x slower than an mmap-ed look up table.

Why ever use mmap instead of sharded inverted indices of word-doc here, a la elasticsearch?

Yeah the question is what level of performance you need I guess... was hoping you could clarify :)

But you don't actually need that level of performance? You've made this system more complex and expensive to achieve a requirement that doesn't matter?

you seem to have a deeper knowledge of the business & organisational context that dictate the true requirements than someone working there. please share these details so we can all learn!

Sure: the network request time of a person making a request over the open internet is going to be an order of magnitude longer than a DB lookup (in the right style, with a reverse-index) on the scale of data this person is describing. So making the lookup 10x faster saves you...1% of the request latency.

And at the qps they've described, it's not a throughput issue either. So I'm pretty confident in saying that this is a case of premature optimization.

And at some point the increase in parallelization of scans dominates mmap speed, unless you're redundantly sharding your mmaped hash table across multiple machines. And there are cases where network bandwidth is the bottleneck before disk bandwidth, though probably not this case. But yeah basically, the answer is something like "if this is the optimal choice, it probably didnt matter that much".

This reads to me as if you have never really used mmap in a dedicated C/C++ application. Just to give you a data point, looking up one word_id in the LUT and reading 20 document_ids from it takes on average 0.0000015 ms.

So if that alternative database takes on average 0.1ms per index read, then it's starting out roughly 65000x slower.

"than a DB lookup (in the right style, with a reverse-index)"

Unless, of course, you're managing petabytes of data ;)

"at the qps they've described, it's not a throughput issue either"

It's mostly a cost thing. If a single request takes 2x the time, that's also a 2x on the hosting bill.

"parallelization of scans dominates mmap speed"

Yes, eventually that might happen. Roughly when you have 100000 servers. But before that your 10gbit/s node-to-node link will saturate. Oops.

> Unless, of course, you're managing petabytes of data ;)

Are...are you saying that you've purchased petabyte(s) of RAM, and that that multi-million dollar investment is somehow cheaper than...well really anything else?

> But before that your 10gbit/s node-to-node link will saturate. Oops.

Only if you're returning dense results, which it sounds like you aren't (and there are ways to address this anyhow), which is why I said the issue of saturating network before disk probably wasn't an issue for you ;)

No, of course I have a tiered architecture. HDDs + SSDs + RAM. By mmap-ing the file, the Linux kernel will make sure that whatever data I access is in RAM and it'll do best-effort pre-reading and caching, which works very well.

BTW, this is precisely how "real databases" also handle their storage IO internally. So all of the performance cost I have to pay here, they have to pay, too.

But the key difference is that with a regular database and indices, the database needs to be able to handle read and write loads, which leads to all sorts of undesirable trade-offs for their indices. I can use a mathematically perfect index if I split dataset generation off of dataset hosting.

It's really quite difficult to explain, so I'll just redirect you to the algorithms. A regular database will typically use a B-tree index, which is O(log(N)). I'm using a direct hash bucket look-up, which is O(1).

For a mental model, you can think of "mmap" as "all the results are already in RAM, you just need to read the correct variable". There is no network connection, no SQL parsing, no query planning, no index scan, no data retrieval. All those steps would just consume unnecessary RAM bandwidth and CPU usage. So where a proper DB needs 1000+ CPU cycles, I might get away with just 1.

No modern DB uses mmap because it's unreliable and hard to tune for performance.

A custom cache manager will always perform better than mmap provided by the kernel.

The problem is you haven't explained how the overhead of a DB is too much. Sure, it sounds like a lot of work for your servers and the DB compared to reading from a hashmap.

Where I work right now we fire around 1.5B queries a day... to Mongo.

And have your unreliable, iconsistent, unscalable system. That apparently goes down all the time.

Not using ES here is actually nuts.

Are you managing petabytes of data though?

What kind of servers are you running? What's your max QPS?

The fact is with your mmap impl. you probably use ram + virtual memory, and have more ram than needed to compensate for the fact that you don't keep the most used keys in memory, which a DB will do for you.

Point is if you have petabytes of data and access patterns only mean you access a subset of it, even Mongo might be cheaper to run.

Just FYI, MongoDB storage also uses mmap internally.

So we are comparing here "just mmap" with "mmap + all that connection handling, query parsing, JSON formatting, buffering, indexing, whatever stuff that MongoDb does".

And no, MongoDB is effectively never a cheap solution. They are used because they are super convenient to work with, with all things being JSON documents. But all that conversion to and from JSON comes at a price. It'll eat up 1000s of CPU cycles just to read a single document. With raw mmap, you could read 1000s of documents instead.

MongoDB uses the Wired Tiger storage engine internally. The MMAP storage engine was removed from MongoDB in V4.2 which was released in March 2020. The MMAP engine was deprecated two years previously.

In MongoDB conversion to and from raw JSON into BSON (Binary JSON) is done on the client (aka driver) so the server cycles are not consumed.

And 2 you're looking past the point. Any DB work work fine for this use case. If you wanted sharding, there's vitess for mysql, for example.

As another already said, Mongo doesn't use mmap anymore.

Mongo doesn't convert to and from JSON. The driver uses a binary protocol.

As a security guy I HATE the loss of visibility in going to the cloud. Can you duplicate it? Sure. Still not as easily as spanning a trunk and you still have to trust what you’re seeing to an extent.

The visibility I was mentioning in the parent comment was visibility from executives in your business, but I can see how it would be confusing.

There are tradeoffs — cloud removes much of the physical security risks and gives you tools to help automated incident detection. Things like serverless functions let you build out security scaffolding pretty easily.

But in exchange you do have to give some trust. And I totally understand resistance there.

> cloud removes much of the physical security risks

Doesn't cloud increase the physical security risks, rather than decrease/remove?

You might be surprised. The performance equivalent of $100k monthly in EC2 spend fits into a 16m2 cage with 52HU racks.

Which costs you more than $100k monthly to operate with the same level of manageability and reliability.

We don't use AWS, because our use cases don't require that level of reliability and we simply cannot afford it, but if I needed a company to depend on IT that generates enough revenue... I probably wouldn't argue about the AWS bill. So long, prepaid at hetzner + in-house works good enough, but I know what I cannot offer with the click of a button to my user!

This is a religious debate among many. The IT/engineering nerd stuff doesn’t matter at all. Cloud migration decisions are always made by accounting and tax factors.

I run two critical apps, one on-prem and one cloud. There is no difference in people cost, and the cloud service costs about 20% more on the infrastructure side. We went cloud because customer uptake was unknown and making capital investments didn’t make sense.

I’ve had a few scenarios where we’ve moved workloads from cloud to on-prem and reverse. These things are tools and it doesn’t pay to be dogmatic.

> These things are tools and it doesn’t pay to be dogmatic.

I wish I would hear this line more often.

So many things today are (pseudo-) religious now. The right frsmework/language, cloud or on prem, x vs not x.

Especially bad imho when somebody tries to tell you how you could do better with 'not x' instead of x you are currently using without even trying to understand the context this decision resides in.

[Edit] typo

> So many things today are (pseudo-) religious now. The right frsmework/language, cloud or on prem, x vs not x.

Might have always been that way? We just have so many more tools to argue over now.

that cage is a liability, not an asset. How is the networking in that rack? What's its connection to large-scale storage (IE, petabytes, since that's what I work with). What happens if a meteor hits the cage? Etc.

That depends on what contracts you have. You could have multiple of these cages in different locations. Also, 1 PB is only 56 large enterprise HDDs. So you just put storage into the cage, too.

But my point wasn't about how precisely the hardware is managed. My point was that with a large cloud, a mid-sized company has effectively NO SUPPORT. So anything that gives you more control is an improvement.

"1 pb is only 56 large enterprise hdds".

umm, what happens when one fails?

With large cloud my startup had excellent support. We negotiated a contract. That's how it works.

Typically people use RAID or ZFS to prevent data loss when a few hdds fail.

OK, so basically you're in a completely different class of expectations about how systems perform under disk loss and heavy load then me. A drive array is very different from large-scale cloud storage.

Hard to say. My impression is:

- A large ZFS pool of SSDs is much faster than any cloud storage.

- Cloud storage failed much more often than the SSDs in our pool.

- "Noisy neighbor" is an issue on the cloud

This cracked me up. Thanks fxtentacle :D.

of course, the reason that's wrong is that if one drive fails you don't have a 56pb storage system, you have something smaller because of redundancy.

That redundancy, and the performance that scales due to it, place cloud services in an entirely different class from on prem servers.