Is it possible that you haven't found memory safety issues because you haven't been looking for them, haven't been fuzz testing, or haven't had adversarial users of your codebases?

These are database kernels used for mission-critical things with extreme workload profiles. It has to survive longevity testing at saturating workloads with continuous fault injection. If there were memory safety issues, I think someone would have noticed by now.

The reality is that there isn’t much opportunity for memory safety issues to occur anyway, the type system and scheduler do most of the heavy lifting. Similarly, concurrency safety isn’t much of an issue because threads barely interact. Most high-performance server software looks this way these days.

Bugs tend to be of the boring logic variety that can happen in any programming language.

> These are database kernels used for mission-critical things with extreme workload profiles.

> Similarly, concurrency safety isn’t much of an issue because threads barely interact.

I think the domain you're working in isn't as susceptible to memory safety issues, but that doesn't mean they're not present. It also sounds like they domain you're working in is trivially parallelized if threads "barely interact", which limits your exposure to those memory and data race issues.

If you gave an adversary access to the API of your kernel however, how long do you think before they found a use-after-free, double-free, stack or heap overflow, etc? Days, weeks, or hours?

If you haven't run a fuzzer yet, I wouldn't be so confident.

I assure you that the people working on web browsers are in general quite competent and even so often have trouble with memory safety.

If you are able to structure your code so there is only ever one construction order for all objects your program uses, and destruction is the reverse order, you're pretty safe for most memory safety bugs.

Sadly as codebases get more complex, that vision gets further and further from reality.

Do you fuzz your code or use something like asan/msan (https://github.com/google/sanitizers)? If not, how do you know that your code is really bug free?

Far more exhaustive testing than that, these are database kernels. Formal verification of the design and code where it makes sense. Extensive use of the C++ type infrastructure to verify several aspects of correctness at compile-time that would be impractical in any other systems language. The Linux kernel is bypassed for almost everything, most of the schedulers, I/O memory management, etc is in user space like most modern high-performance databases, which actually reduces the bug surface and increases verifiability. It doesn’t have to just be functionally correct, the performance has to be consistent under diverse and adversarial workloads.

No one can know if it is bug free, that is impractical. But it also isn’t like this is a weekend hobby project either. Most of the bugs that get out are in unimportant peripheral code and integrations.

Ok then a followup question: do you use pointers in your code and allocate objects on the heap?

I'm quite curious what tools you're using to formally verify your C++ code if you are. My understanding is that in general you can't, which is why msan/asan exist, to get a first approximation of verifying things that can't be formally verified for most C++ code.

(In general, I'm dubious of these claims that "Most serious projects won’t hire you if you aren’t capable of writing memory safe code in your sleep", because I think if you asked the majority of the members of the C++ committee if they could do that, they'd say no).

Database engines typically operate within a giant block of memory allocated at bootstrap which is used quite differently than a heap. Mutable references to the address space necessarily exist outside the address space, which presents problems for conventional memory safety models. There are also no pointers per se because almost all memory is directly paged -- objects have no fixed memory address over their lifetime. Most shops use a thin wrapper that implements a pointer-like interface (in the style of std::unique_ptr) that hides the paging and scheduling mechanics. The scheduler design, which is where most of the safety and guarantees that resources can't leak happens, is often verified with a model checker. It is perfectly safe to hold an arbitrary number of concurrent mutable references to an object because safe execution will be resolved dynamically at runtime (which is cheaper than it sounds). There are container libraries designed to work seamlessly in this type of memory model. To a developer writing code against this abstraction, it feels similar to a garbage-collected concurrency-safe language. No locking, no resource management. The abstraction isn't general purpose but it works very well for the kinds of data infrastructure applications where people use C++.

In many of these systems it is standard practice to generate arithmetically limited types pervasively. This is almost transparent in C++17. While it is possible to verify much of this at compile-time in theory, it almost never is because it isn't worth the effort (C++20 may start to change this) and testing at runtime has proven to be nearly as good. People underestimate what is possible with the C++ type infrastructure in this regard.

This type of software design was originally done because it allows for exceptional performance but has become popular for safety reasons. It uniquely allows you to make guarantees about runtime behavior under diverse adversarial workloads that would otherwise be difficult to make.

Bugs in practice tend to occur at the interface with third-party code, which requires dropping out of any internal type system, or in the form of performance anomalies due to unexpected hardware behaviors interacting with the scheduler design. Logic bugs in the core bits tend to be found in testing.

Right, so you're working in a very particular style of application. The style of programming you're using doesn't work for most software and dodges the source of some memory safety issues.

Presuming that this style works equally well for all software seems presumptuous and perhaps naive, does it not?

Do you know of any open-source code architected in the way you explain above. Have you looked at Scylladb codebase, do you know if it is architected this way.