However, they (at least at the time most of the developers were at Apple, many have now moved to Snowflake and the Apple team has grown a little I think) haven't released or integrated their nightly cluster and performance testing systems into open, nor have they integrated them with GitHub Actions or Nightly runs or anything. So while Apple or Snowflake integrates changes across hundreds of concurrent fuzzing simulations on whatever platforms they have, if you write patches yourself, you're stuck with long simulation runs. Maybe that's changed; I haven't kept up since the 7.0 series.
In practice if you write patches and they accept them, they will just do the testing in their runs for you, on a cluster far larger than what you could have.
Someone could probably integrate this functionality into a Kubernetes operator or something so that outside engineers could run large scale simulations reliably. But it is really expensive and CPU compute intensive no matter how you go about it.
[1] https://forums.foundationdb.org/t/how-to-use-foundationdb-un...
https://github.com/apple/foundationdb/tree/main/fdbserver/wo...
> Someone could probably integrate this functionality into a Kubernetes operator or something so that outside engineers could run large scale simulations reliably. But it is really expensive and CPU/compute intense, no matter how you go about it.
Maybe this.
And yes, I linked to the spec files because there actually isn't that much test code written in Flow I feel; the high-level specs in the .txt files can be mixed and matched so much to create a lot of variety from some small number of primitives, so that's really where all the good stuff is. Implementation vs interface, and all that.
It’s not a full-blown simulator (because generally the application code doesn’t even exist yet when you’re building the TLA+ model). But it can let you collect data and validate assumptions about your design before writing a single line of code.
In reality the design stage is a pretty critical phase so you need all the help you can get, so even if you don't like TLA+ you're way better off than not modeling at all.
As an example of the language specific thing, though, there's a library for Haskell I like that's very cool, called Spectacle, which also implements the temporal logic of TLA+ along with a model checker, but as a Haskell DSL. An interesting benefit of this is that you can model check actual real Haskell code that runs e.g. in your services, but I haven't taken this very far. There are also alternative solutions like Stateright for Rust. But again, not everyone has the benefit of these...
Unfortunately I got the product wrong; it was not Atlas, it was Realm Sync. All of the test-case generation stuff is in Section 5.
Documentation is still generally useful, and so is a model. You have to be committed to keeping both up to date as the code evolves.
Using https://github.com/awslabs/shuttle which works on our real Rust code.
This is such an interesting topic!
Some thoughts:
* I wonder if the approach could be used to implement debuggable replayability, with accurate tracing and profiling. A bit like what verdagon is doing with Vale.
* It could be used to integrate the event loop with tracing (rather than instrumentation with Jaegar)
* I really like the idea that "every object" is an event loop, which reminds me of Microsoft Orleans with its actor model for its grains.
* I am interested with actor and lightweight thread architectures.
* I am interested in the scalabiliy of nodejs event loop architecture and Win32 desktop application programming.
* I think this approach could be used to test and simulate microservices.
* Approach could be used to test GUIs with React Redux reducer style.
https://github.com/AaronFriel/hyhac/blob/master/test/Test/Hy...
Alas, I think Hyperdex development paused a few years later. It's a shame that it stopped then.
https://www.youtube.com/watch?v=4fFDFbi3toc
> We wanted FoundationDB to survive failures of machines, networks, disks, clocks, racks, data centers, file systems, etc., so we created a simulation framework closely tied to Flow. By replacing physical interfaces with shims, replacing the main epoll-based run loop with a time-based simulation, and running multiple logical processes as concurrent Flow Actors, Simulation is able to conduct a deterministic simulation of an entire FoundationDB cluster within a single-thread! Even better, we are able to execute this simulation in a deterministic way, enabling us to reproduce problems and add instrumentation ex post facto. This incredible capability enabled us to build FoundationDB exclusively in simulation for the first 18 months and ensure exceptional fault tolerance long before it sent its first real network packet. For a database with as strong a contract as the FoundationDB, testing is crucial, and over the years we have run the equivalent of a trillion CPU-hours of simulated stress testing.
https://pierrezemb.fr/posts/notes-about-foundationdb/