News for hackers. Stuff that matters.
Re:Jepsen: MongoDB 4.2.6 by aphyr on Sunday May 24, 2020 @08:12AM (#23290948)
Hi folks! Author of the report here. If anyone has questions about detecting transactional anomalies, what those anomalies are in the first place, snapshot isolation, etc., I'm happy to answer as best I can.
Re:Jepsen: MongoDB 4.2.6 by devit on Sunday May 24, 2020 @10:35AM (#23291800)
Have you considered presenting the data in a concise manner in addition to the in-depth analyses?

That is, a table on the jepsen.io frontpage, or at least on each product's review page, with database products and configuration on rows and consistency properties on columns, and a nice "Yay!" or "Nope!" mark in the cell, plus links on how to achieve the database configurations in the table (esp. how to configure each database to have the most guarantees).

Also, ideally the analyses should be rerun automatically (or possibly after being paid, but making it easy for the company to do so) every time a new major release happens rather than being done once and then being stale.

Finally, there should be tests for the non-broken databases (PostgreSQL for instance, both in single-server mode, deployed with Stolon on Kubernetes and using the multimaster projects) as well to confirm they actually work.

Re:Jepsen: MongoDB 4.2.6 by aphyr on Sunday May 24, 2020 @10:45AM (#23291847)
That is, a table on the jepsen.io frontpage, or at least on each product's review page, with database products and configuration on rows and consistency properties on columns, and a nice "Yay!" or "Nope!" mark in the cell, plus links on how to achieve the database configurations in the table (esp. how to configure each database to have the most guarantees).

This is a wonderful idea, and I've got no idea how to actually do it in a standardized, rigorous way. Vendor claims are often contradictory, it's hard to get a good idea of anomaly frequency, availability is... a rabbithole, and it's hard to come up with a standard taxonomy of anomalies--most of the analyses I do wind up finding something I've never really seen before, haha. With that in mind, I've wound up letting the reports speak for themselves.

Also, ideally the analyses should be rerun automatically (or possibly after being paid, but making it easy for the company to do so) every time a new major release happens rather than being done once and then being stale.

I don't know a good way to do this either. Each report is typically the product of months of experimental work; it's not like Jepsen is a pass-fail test suite that gives immediately accurate results. There is, unfortunately, a lot of subtle interpretive work that goes into figuring out if a test is doing something meaningful, and a lot of that work needs to be repeated on each test run. Think, like... staring at the logs and noticing that a certain class of exception is being caught more often than you might have expected, and realizing that a certain type of transaction now triggers a new conflict detection mechanism which causes higher probabilities of aborts; those aborts reduce the frequency with which you can observe database state, allowing a race condition to go un-noticed. That kinda thing.

If I'm lucky and the API/setup process haven't changed, I can re-run an analysis in about a week or so. If I'm unlucky, there's been drift in the OS, setup process, APIs, client libraries, error handling, etc. It's not uncommon for a repeat analysis to take months. :-(

Re:Jepsen: MongoDB 4.2.6 by X6S1x6Okd1st on Sunday May 24, 2020 @11:24AM (#23292145)
It's probably more snarky than helpful, but it'd be great to have a section where it's just marketing materials or docs that you've corrected with a red pen
Re:Jepsen: MongoDB 4.2.6 by bcrosby95 on Sunday May 24, 2020 @11:47AM (#23292328)
It's probably better to keep it professional. Your average employee can afford some snark. But when companies hire you for this sort of consulting, you could turn off a lot of potential clients by including it in materials you produce, even when they didn't pay for it. Because it is a representation of the product they would be paying for.

It would be kinda like you including this sort of thing on your resume. Which would also be a bad idea.

Re:Jepsen: MongoDB 4.2.6 by ashtonkem on Sunday May 24, 2020 @04:45PM (#23294534)
For those who don’t know, Kyle makes a living offering these types of analysis to database companies directly. While a lot of us love to dunk on Mongo (myself included), it would be silly to expect Kyle to risk his livelihood.
Re:Jepsen: MongoDB 4.2.6 by jka on Sunday May 24, 2020 @11:55AM (#23292375)
If done accurately and professionally, something like you're suggesting could be really useful to aid people and organizations during vendor selection.

https://web.hypothes.is/about/ or similar could be used to develop commentary overlays on top of marketing materials.

Re:Jepsen: MongoDB 4.2.6 by HappyDreamer on Sunday May 24, 2020 @10:49AM (#23291878)
> consistency properties on columns, and a nice "Yay!" or "Nope!" mark in the cell

Plus maybe a column indicating what [the company behind the database] claims?

Re:Jepsen: MongoDB 4.2.6 by eloff on Sunday May 24, 2020 @10:44AM (#23291845)
Oh man this would be useful.
Re:Jepsen: MongoDB 4.2.6 by politician on Sunday May 24, 2020 @11:20AM (#23292105)
Thank you for all of your work over the years. Your reports have helped me and others stand up to bizdev hype and make better decisions for our companies and customers.

Postgres is widely understood to be a robust database with safe defaults. I, and perhaps others, would love to see you aim your array of weapons at Postgres. Do you have any plans to look at stock Postgres?

Re:Jepsen: MongoDB 4.2.6 by aphyr on Sunday May 24, 2020 @11:37AM (#23292247)
It's been on my list for a long time, but I've also struggled to find out like... what, exactly, is the right way to do postgres replication? Every time I go into the docs I wind up with a laundry list of different mechanisms for replication and failover, and no idea which one would be most appropriate for a test. I gotta get on this!
Re:Jepsen: MongoDB 4.2.6 by zbjornson on Sunday May 24, 2020 @12:33PM (#23292632)
It'd be especially interesting given that MongoDB claims this:

> Postgres has both asynchronous (the default) and synchronous replication options, neither of which offers automatic failure detection and failover [12]. The synchronous replication only waits for durability on one additional node, regardless of how many nodes exist [13]. Additionally, Postgres allows one to tune these durability behaviors at the user level. When reading from a node, there is no way to specify the durability or recency of the data read. A query may return data that is subsequently lost. Additionally, Postgres does not guarantee clients can read their own writes across nodes.

From http://www.vldb.org/pvldb/vol12/p2071-schultz.pdf

Re:Jepsen: MongoDB 4.2.6 by takeda on Sunday May 24, 2020 @02:48PM (#23293655)
> It'd be especially interesting given that MongoDB claims this:

> > Postgres has both asynchronous (the default) and synchronous replication options, neither of which offers automatic failure detection and failover [12]. The synchronous replication only waits for durability on one additional node, regardless of how many nodes exist [13]. Additionally, Postgres allows one to tune these durability behaviors at the user level. When reading from a node, there is no way to specify the durability or recency of the data read. A query may return data that is subsequently lost. Additionally, Postgres does not guarantee clients can read their own writes across nodes.

> From http://www.vldb.org/pvldb/vol12/p2071-schultz.pdf

This is like those commonly seen tables comparing your product with others where your product had checkmarks in all categories, and of course competitors are missing a bunch of them. The problem is that the categories were picked by you, and are often irrelevant to the other product. This is the case here.

PostgreSQL is not a distributed database, the master is the one doing all writes. The replicas are read only. By default replicas are asynchronous which means they won't affect master performance, at the cost of having data there being late by few seconds. Since you can't write to replicas, this won't cause data corruption, only delay which often is acceptable. If you design your applications in such way that will have two database endpoints: one for writes and one just for reads, you can then decide based on context which endpoint you want to use. The read only is easy to scale, but as mentioned earlier it is read only, and might slight delay.

Now, for failover, you might also opt on using synchronous replicas this will add extra latency, but then you always have at least one machine that has the same data. They mentioned that if you have multiple synchronous standbys then it only one needs to write. Actually that's configurable, you can specify group of synchronous machines and how many and which need to be synchronized, the remaining ones are a backup in case those that you specified aren't available.

Besides, the writes don't work the same way as in mongo, when a standby node is in sync it isn't just in sync for that particular write, it is completely in sync, so their following argument about not being able to specify durability/recency of data on read is redundant. If you contact the master or synchronous replica, you will always get the most recent state. If you don't mind slight delay you should query asynchronous replicas (in fact you should prefer them whenever you can, since those are cheap to add)

Re:Jepsen: MongoDB 4.2.6 by zbjornson on Sunday May 24, 2020 @03:16PM (#23293879)
I'm not sure I understand your point.

> the master is the one doing all writes. The replicas are read only. By default replicas are asynchronous

The same is true with MongoDB's defaults in an unsharded cluster.

Re:Jepsen: MongoDB 4.2.6 by pcl on Sunday May 24, 2020 @02:14PM (#23293384)
I think that it'd be super-valuable to do an analysis of an RDS Postgres deployment. Amazon is doing some dark magic with RDS that sits at this really interesting "distributed, but not that distributed" inflection point, which impacts the basic assumptions of lots of distributed database design.

I believe RDS Postgres is probably the right answer for lots of applications, especially for those that already depend on AWS for baseline availability. I'd love to see if that holds up against a rigorous analysis.

Re:Jepsen: MongoDB 4.2.6 by aphyr on Sunday May 24, 2020 @02:40PM (#23293579)
I'd like this too, but I'm not sure how to do fault injection against an Amazon-controlled service.
Re:Jepsen: MongoDB 4.2.6 by ashtonkem on Sunday May 24, 2020 @04:47PM (#23294548)
You’d probably have to work directly with AWS on that one, either to get a custom harness in AWS, or to find out how they configure RDS replication.
Re:Jepsen: MongoDB 4.2.6 by elesbao on Sunday May 24, 2020 @05:21PM (#23294936)
the setup would prob ably be pgsql primary, aurora secondaries on diff zones and something changing cross-zone or cross-region vpc setting to try to break replication ? never tried that but was hurt by rds pure pgsql cross region replication in a network outage situation.
Re:Jepsen: MongoDB 4.2.6 by aeyes on Sunday May 24, 2020 @02:27PM (#23293469)
Are you talking about Aurora? Because in RDS the replication is just what you get out of the box.
Re:Jepsen: MongoDB 4.2.6 by bsaul on Sunday May 24, 2020 @12:24PM (#23292579)
i feel like this is the reaction of everyone having ever tried to setup postgres replication. With your audience, you deciding for a particular setup will probably help a LOT of people, and ultimately the postgres project as well.
Re:Jepsen: MongoDB 4.2.6 by takeda on Sunday May 24, 2020 @02:11PM (#23293355)
If you worry about data, you should not use automatic failover. It's nearly impossible for standby to know why master stopped responding. Maybe there was a hardware failure, or maybe master is just busy. This is why manual failover is better, because you can know the real reason and decide whether you should perform failover or just wait.

With tools like repmgr it is just a single command invoked on the standby.

If you absolutely don't want to lose any data, you should have two masters in close proximity (so the latency isn't high) set up with synchronous replication, then have one or two standbys with asynchronous replication. This will reduce throughout, but then you can be sure that the other machine has all the same transactions. If something happens to both you then can fallback to the asynchronous one which might be a bit behind.

Re:Jepsen: MongoDB 4.2.6 by feike on Sunday May 24, 2020 @03:15PM (#23293863)
One of the authors of Patroni here.

Automatic failover for PostgreSQL works great and can be done safely if combined with synchronous replication.

Multiple tools will implement this correctly:

https://patroni.readthedocs.io/en/latest/replication_modes.h... https://github.com/sorintlab/stolon/blob/master/doc/syncrepl...

Quoting a former colleague here, but "if it hurts, do it more often". That is what you should do with your PostgreSQL failovers.

I have clusters running on timelines in the hundreds without a byte of data loss due to using synchronous replication, tools that help out with leader election, and just doing it often.

Re:Jepsen: MongoDB 4.2.6 by takeda on Sunday May 24, 2020 @06:37PM (#23295522)
Can Patroni tell if master node is not responsive because it is busy vs dead? GitHub (I believe) had few outages that caused data loss because their auto failover mechanism kicked in when it shouldn't.

I would actually be interested if aphyr's analysis of Patroni and other distributed add-ons to PostgreSQL.

Re:Jepsen: MongoDB 4.2.6 by pas on Sunday May 24, 2020 @08:58PM (#23296574)
There is no real difference between dead or too-busy.

The only question is how soon are you going to page humans. After the automated mechanism flipped your master 2-3 times but the cluster still hasn't made progress [nothing coming out of the master; or it locks up after a few minutes again]), or right after some other automated mechanism detects that there's a problem.

Whatever automation you have in place, it has advantages and disadvantages. In the GitHub case - I suppose - they determined post-mortem that it would have been better to just let the master chug through the incoming onslaught of queries instead of failing over, and over, and over. (But of course this seems like a trivial problem in any auto failover setup, so I suspect there's more to the story.)

Re:Jepsen: MongoDB 4.2.6 by feike on Monday May 25, 2020 @05:10AM (#23299395)
> Can Patroni tell if master node is not responsive because it is busy vs dead

No. But the contract Patroni has is this:

I only serve a master (primary) if I have the lock. If I do not have the lock I will demote.

This results in that there can be only 1 primary active at any given point in time, even if the network is partitioned.

This in and of itself does not guarantee no-split-brain situations, a split-brain can occur if writes were made on the former primary, but not yet on the future primary. This however can be mitigated with synchronous replication.

Re:Jepsen: MongoDB 4.2.6 by zozbot234 on Sunday May 24, 2020 @09:30PM (#23296806)
> tell if master node is not responsive because it is busy vs dead?

The postgres documentation will tell you that you'll need to set up your own mechanisms for this, and that they will need to integrate with OS facilities as appropriate. One-size-fits-all does not cut it. Not wrt. replication, not wrt. HA/failover.

Re:Jepsen: MongoDB 4.2.6 by takeda on Sunday May 24, 2020 @01:57PM (#23293242)
Well, the built-in ways is the right way to do it. But given that PostgreSQL is quite conservative about it, it will be hard to find issues there (the replicas are read only, so at worst it will be just a replication delay, unless you use synchronous replication, which will remove the replication delay at the cost of slower performance).

All the tooling that provides extra distributed functionality not present in postgres (auto failover, multi master replication, sharding etc) will surely have issues, but then you aren't testing the PostgreSQL itself, but the tooling, so to be fair, you the article should evaluate these tools, and any shortcomings shouldn't go to PostgreSQL (unless it really is a PostgreSQL issue).

Re:Jepsen: MongoDB 4.2.6 by pas on Sunday May 24, 2020 @08:51PM (#23296509)
This k8s operator advertises production-grade auto-failover auto-healing replicas - basically what MongoDB - but built on postgres: https://github.com/CrunchyData/postgres-operator

And looking at this table, basically the future seems to be WAL shipping anyway ( https://www.postgresql.org/docs/13/different-replication-sol... )

Re:Jepsen: MongoDB 4.2.6 by didip on Sunday May 24, 2020 @04:09PM (#23294309)
It is true that only recently PG has a standard way of replicating. But even then, PG is not a distributed database by default.

However, if I may suggest, Stolon, Patroni, Postgres XL or Citus Data might be interesting to you.

Re:Jepsen: MongoDB 4.2.6 by grogers on Monday May 25, 2020 @11:26AM (#23302010)
I know you've done galera in the past, but MySQL group replication would be a good one since it IS the defacto HA solution, unlike with Postgres.
Re:Jepsen: MongoDB 4.2.6 by zzzcpan on Sunday May 24, 2020 @11:42AM (#23292285)
Postgres is not a distributed database and doesn't have a single safe default for running it in a distributed configuration, including talking to it over network. It can't claim any consistency guarantee, so there is nothing for aphyr to test it for.

Even common highly available configurations take the route of no consistency guarantees by doing primitive async replication and primitive failover.

Re:Jepsen: MongoDB 4.2.6 by politician on Sunday May 24, 2020 @11:58AM (#23292391)
Postgres supports multi-master replication, among other replication models. This could provide an interesting target.

In a classic single node configuration, a confirmation that its transaction isolation behaviors exhibited the corresponding anomalies would be valuable.

So I think there’s value in this ask.

Re:Jepsen: MongoDB 4.2.6 by samdk on Sunday May 24, 2020 @01:14PM (#23292945)
Postgres doesn't natively support multi-master. (Although there are a variety of open source/proprietary offerings that add support for it to various degrees.)
Re:Jepsen: MongoDB 4.2.6 by takeda on Sunday May 24, 2020 @02:52PM (#23293679)
PostgreSQL doesn't offer multi master replication. There are extensions that do, but if aphyr will evaluate them he should emphasize that he is treating them not the PostgreSQL (unless he finds a bug in PostgreSQL itself).

I think he did something similar for MySQL when evaluating the Galera cluster.

Re:Jepsen: MongoDB 4.2.6 by politician on Sunday May 24, 2020 @03:18PM (#23293893)
Jepsen reports often include two distinct types of analyses: correctness in a distributed storage system under a variety of failure scenarios, and in-depth analysis of consistency claims. Both examinations are extremely helpful.

In a single write master configuration, Postgres runs transactions concurrently, so the consistency analysis is still quite relevant.

I don’t think it’s a stretch to say that everyone expects Postgres to get top marks in this configuration and it would be worth confirming that this is the case.

Re:Jepsen: MongoDB 4.2.6 by takeda on Sunday May 24, 2020 @03:42PM (#23294087)
Actually he already did analyze PostgreSQL: https://aphyr.com/posts/282-call-me-maybe-postgres

But it was long ago, and maybe needs to be redone?

Edit: after re-reading it he treats it as a distributed system because client and server is over network. And that is true, it can also be thought of as a distributed system because as you said transactions are concurrent and are running as separate processes. Although in these cases you can't have a partition (which aphyr uses to find weaknesses), or maybe there is something equivalent that happens?

Re:Jepsen: MongoDB 4.2.6 by zozbot234 on Sunday May 24, 2020 @03:19PM (#23293907)
> PostgreSQL doesn't offer multi master replication.

Not in itself, but it does offer a PREPARE TRANSACTION - COMMIT PREPARED / ROLLBACK PREPARED extension that could be used to add such support in the future. This would not be unprecedented, as the simpler case of db sharding is already being supported via the PARTITION BY feature, combined with "FOREIGN" database access.

Re:Jepsen: MongoDB 4.2.6 by bsaul on Sunday May 24, 2020 @12:28PM (#23292599)
i'm not sure what you mean by pg not being a "distributed" database. it has replication and sharding functionalities that let it run in various clustering configuration. This looks enough to me to qualify it for aphyr tests.
Re:Jepsen: MongoDB 4.2.6 by takeda on Sunday May 24, 2020 @04:55PM (#23294650)
Replication is read only, so at worst there's only delay when it is set up asynchronously, but ultimately it will be the same as master. The sharding part, do you mean FDW? I don't think PostgreSQL gives any consistency guarantees if you use them.
Re:Jepsen: MongoDB 4.2.6 by bsaul on Sunday May 24, 2020 @05:38PM (#23295047)
ha, my bad. I had the feeling pg provided some solution for sharding, but it seems they're all third-party extensions ( like citus/pg-shard)
Re:Jepsen: MongoDB 4.2.6 by danpalmer on Sunday May 24, 2020 @08:45AM (#23291115)
Not a question necessarily about the technical side, but I'm interested in your opinion as to the root cause – is it desire to achieve certain results for marketing purposes, lack of understanding/training in the team about distributed systems, just bugs and a lack of testing...? Alternatively does most of this come down to one specific technical choice, and why might they have made that choice?

Very happy for (informed) speculation here, I recognise we'll probably never know for certain, but I'm interested to avoid making similar mistakes myself.

Re:Jepsen: MongoDB 4.2.6 by aphyr on Sunday May 24, 2020 @09:05AM (#23291216)
There's a few things at play here. One is talking only about the positive results from the previous Jepsen analysis, while not discussing the negative ones. Vendors often try to represent findings in the most positive light, but this was a particularly extreme case. Not discussing default behavior is a significant oversight, and it's especially important given ~80% of people run with default write concern, and 99% run with default read concern.

The middle part of the report talks about unexpected but (almost all) documented behavior around read and write concern for transactions. I don't want to conjecture too much about motivations here, but based on my professional experience with a few dozen databases, and surveys of colleagues, I termed it "surprising". The fact that there's explicit documentation for what I'd consider Counterintuitive API Design suggests that this is something MongoDB engineers considered, and possibly debated, internally.

The final part of the report talks about what I'm pretty sure are bugs. I'm strongly suspicious of the retry mechanism: it's possible that an idempotency token doesn't exist, isn't properly used, or that MongoDB's client or server layers are improperly interpreting an indeterminate failure as a determinate one. It seems possible that all 4 phenomena we observed stem from the retry mechanism, but as discussed in the report, it's not entirely clear that's the case.

Re:Jepsen: MongoDB 4.2.6 by danpalmer on Sunday May 24, 2020 @09:16AM (#23291276)
Thanks for the thoughts.

I get the impression that MongoDB may have hyped themselves into a corner in the early days with poorly made (or misleading) benchmarks. Perhaps they have customers with a lot of influence determining how they think about performance vs consistency.

Maybe this combined with patching, re-patching, re-patching again their replication logic/consistency algorithm means that they'll be stuck in this sort of position for a long time.

Re:Jepsen: MongoDB 4.2.6 by aphyr on Sunday May 24, 2020 @09:35AM (#23291399)
Possibly! You're right that path dependence played a role in safety issues: the problems we found in 3.4.0-rc3 were related to grafting the new v1 replication protocol onto a system which made assumptions about how v0 behaved. That said, I don't want to discount that MongoDB has made significant improvements over the years. Single-document linearizability was a long time in the works, and that's nothing to sneeze at!

http://jepsen.io/analyses/mongodb-3-4-0-rc3

Re:Jepsen: MongoDB 4.2.6 by eternalban on Sunday May 24, 2020 @08:17AM (#23290969)
"3.4 Duplicate Effects"

This section seems to be the most worrying results in your report, Kyle, with no work around. Did I read that correctly?

Re:Jepsen: MongoDB 4.2.6 by aphyr on Sunday May 24, 2020 @08:27AM (#23291015)
Yeah, there's no workaround that I can find for 3.4 (duplicate effects), 3.5 (read skew), 3.6 (cyclic information flow), or 3.7 (read own future writes). I've arranged those in "increasingly worrying order"--duplicating writes doesn't feel as bad as allowing transactions to mutually observe each other's effects, for example. The fact that you can't even rely on a single transactions' operations taking place (or, more precisely, appearing to take place) in the order they're written is especially worrying. All of these behaviors occurred with read and write concerns set to snapshot/majority.

That's not to say that workarounds don't exist, just that I didn't find any in the documentation or by twiddling config flags in the ~2 weeks I was working on this report. :)

Re:Jepsen: MongoDB 4.2.6 by rystsov on Sunday May 24, 2020 @04:02PM (#23294267)
Hi Kyle, thanks for the Elle :) I want to use Elle to check long histories of transactions over small set of keys with read dominant workload, the paper recommends to use lists over registers but when the history becomes long on the one hand it becomes too wasteful to read the register's history on each request on the other hand the Elle's input becomes very large. E.g. when each read should return the whole register's history the size of history grows O(n^2) compared to the case when the reads return just the head.

So I'm curios how would you have described the ability of finding violations with Elle using read-write registers with unique values vs the append-only lists?

Re:Jepsen: MongoDB 4.2.6 by aphyr on Sunday May 24, 2020 @05:49PM (#23295144)
E.g. when each read should return the whole register's history the size of history grows O(n^2) compared to the case when the reads return just the head.

If you look at Elle's transaction generators, you can cap the size of any individual key, and use an uneven (e.g. exponential) distribution of key choices to get various frequencies. That way keys stay reasonably small (I use 1-10K writes/key), some keys are updated frequently to catch race conditions, and others last hundreds of seconds to catch long-lasting errors.

So I'm curios how would you have described the ability of finding violations with Elle using read-write registers with unique values vs the append-only lists?

RW registers are significantly weaker, though I don't know how to quantify the difference. I've still caught errors with registers, but the grounds for inferring anomalies are a.) less powerful and b.) can only be applied in certain circumstances--we talk about some of these details in the paper.

Re:Jepsen: MongoDB 4.2.6 by monstrado on Sunday May 24, 2020 @09:21AM (#23291311)
Huge fan of your work! I was curious if you've ever attempted to run your (or part of) Mongo test suite against FoundationDB using their DocumentLayer since it's supposed to be Mongo API compatible.
Re:Jepsen: MongoDB 4.2.6 by aphyr on Sunday May 24, 2020 @09:33AM (#23291377)
No, I haven't! You can see a full list of analyses here: http://jepsen.io/analyses
Re:Jepsen: MongoDB 4.2.6 by robterrell on Sunday May 24, 2020 @11:46AM (#23292314)
IIRC one of the FoundationDB engineers tested with Jepsen and found that it passed in its default configuration, but the blog post seems to have disappeared.

https://web.archive.org/web/20150312112556/http://blog.found...

Re:Jepsen: MongoDB 4.2.6 by monstrado on Sunday May 24, 2020 @03:02PM (#23293760)
Thanks for firing up the time machine! I've been using FDB for a little over a year now and can't recommend it enough. Such a solid piece of meticulous engineering.
Re:Jepsen: MongoDB 4.2.6 by teskk123 on Sunday May 24, 2020 @09:20AM (#23291303)
hello :) Where is you found out all information about how to do such testing?
Re:Jepsen: MongoDB 4.2.6 by aphyr on Sunday May 24, 2020 @09:29AM (#23291364)
I was lucky to have a good education: my B.A. involved courses in contemporary experimental physics and independent research in nonlinear quantum dynamics (esp. proofs, experimental design, writing), cognitive and social psychology (more experiment design and stats), math structures (proof techniques), philosophy (metaphysics, philosophy of science), and English (rhetoric). All of those helped give me a foundation for doing this kind of experimental work and communicating it to others.

Jepsen draws inspiration from a long line of work on property-based testing, especially Quickcheck & co. It also draws on roughly 10 years of experience building & running distributed systems in production. A lot of Jepsen I invented from whole cloth, but some of the checkers in Jepsen are derived from specific research papers, like work by Wing, Gong, and Howe on linearizability checking.

Then it's just... a lot of thinking, experimenting, and writing. Jepsen's the product of ~6 years of full-time work. Elle, the system which detected the anomalies in this report, was a research project I've been puzzling over for roughly two years.

I write the Jepsen series, and open-source all of the code for these tests, partly as a resource so that other people can learn to do this same kind of work. :-)

Re:Jepsen: MongoDB 4.2.6 by throwaway_pdp09 on Sunday May 24, 2020 @12:22PM (#23292555)
I guess you've answered my question, but to be clear, you do not instrument/analyse the code, you treat it as a black box which you hammer on externally, is that right?
Re:Jepsen: MongoDB 4.2.6 by aphyr on Sunday May 24, 2020 @02:42PM (#23293605)
Pretty much, yeah. There are some cases where Jepsen reaches into the guts of a database or lies to it via LD_PRELOAD shims, but generally these are Just Plain Old Binaries provided by vendors; no instrumentation required.
Re:Jepsen: MongoDB 4.2.6 by teskk123 on Sunday May 24, 2020 @10:14AM (#23291653)
wow. thanks a lot for quick and full answering!
Re:Jepsen: MongoDB 4.2.6 by rclayton on Sunday May 24, 2020 @03:17PM (#23293885)
Hi Kyle! I’ve really enjoyed your work over the years. I was wondering, with all of your testing and experimentation, is there any system that had really impressed you?
Re:Jepsen: MongoDB 4.2.6 by zbentley on Sunday May 24, 2020 @03:39PM (#23294060)
I don't presume to speak for him, but his writeup on ZooKeeper was among the most positive in the Jepsen series: https://aphyr.com/posts/291-jepsen-zookeeper

My bias: I like and heavily use ZooKeeper in production. HN seems not to like it as much.

Re:Jepsen: MongoDB 4.2.6 by aphyr on Sunday May 24, 2020 @05:49PM (#23295151)
I'm kind of impressed any distributed system gets off the ground. These things are hard to write!
Re:Jepsen: MongoDB 4.2.6 by staticassertion on Sunday May 24, 2020 @02:54PM (#23293695)
I've wanted to try building a toy database to learn more about how they work - any suggestions for good resources?
Re:Jepsen: MongoDB 4.2.6 by aphyr on Sunday May 24, 2020 @08:49PM (#23296495)
I've actually written a little mini-Jepsen workbench for folks who want to practice implementing a distributed key-value store. Might be worth a spin! https://github.com/jepsen-io/maelstrom
Re:Jepsen: MongoDB 4.2.6 by dilandau on Sunday May 24, 2020 @01:03PM (#23292862)
You're doing very, very valuable work. Thanks fam, keep those vendors honest, and help us make informed decisions.