News for hackers. Stuff that matters.
Re:Jepsen: Datomic Pro 1.0.7075 by belter on Wednesday May 15, 2024 @06:02PM (#40372953)
> I was also surprised that Jepsen didn’t find critical bugs.

From the report..."...we can prove the presence of bugs, but not their absence..."

Re:Jepsen: Datomic Pro 1.0.7075 by vasco on Wednesday May 15, 2024 @06:36PM (#40373236)
That's consistent with the usual definition of "finding" anything.
Re:Jepsen: Datomic Pro 1.0.7075 by cdchn on Wednesday May 15, 2024 @08:10PM (#40373850)
"Absence of evidence is not evidence of absence."
Re:Jepsen: Datomic Pro 1.0.7075 by andreareina on Wednesday May 15, 2024 @09:19PM (#40374260)
If you've looked, it is. The more and the better you look, the better evidence it is.
Re:Jepsen: Datomic Pro 1.0.7075 by kelseyfrog on Wednesday May 15, 2024 @09:50PM (#40374456)
If you run it through bayes theorem, it adjusts the posterior very little.
Re:Jepsen: Datomic Pro 1.0.7075 by cwillu on Thursday May 16, 2024 @03:22PM (#40382092)
If a test almost always finds something, then the failure of that test to find something is strong evidence.
Re:Jepsen: Datomic Pro 1.0.7075 by kelseyfrog on Thursday May 16, 2024 @04:06PM (#40382543)
I'd be happy to see you numbers for estimated likelihood, prior, and marginal probability if you have them. I'm curious what you get.
Re:Jepsen: Datomic Pro 1.0.7075 by kelseyfrog on Wednesday May 15, 2024 @09:49PM (#40374447)
Thank you. I've updated my initial guess of p(critical bugs | did not find critical bugs) from 0.5 to 0.82 given my estimate of likelihood and base rates.
Re:Jepsen: Datomic Pro 1.0.7075 by nine_k on Thursday May 16, 2024 @02:25PM (#40381556)
s/evidence/proof/.

Evidence of absence ("we searched really carefully and nothing came up") does update the Bayesian priors significantly, so the probability of absence of bugs can now be estimated as much higher.

Re:Jepsen: Datomic Pro 1.0.7075 by jupp0r on Wednesday May 15, 2024 @09:06PM (#40374191)
In practical terms, if you are a database and Jepsen doesn't find any bugs, that's as much assurance as you are going to get in 2024 short of formal verification.
Re:Jepsen: Datomic Pro 1.0.7075 by pests on Wednesday May 15, 2024 @11:58PM (#40375070)
The work antithesis has been doing here has me really excited as well.
Re:Jepsen: Datomic Pro 1.0.7075 by stuarthalloway on Thursday May 16, 2024 @11:19AM (#40379472)
Formal verification is very powerful but still not full assurance. Fun fact: Testing and monitoring of Datomic has sometimes uncovered design flaws in underlying storages that formal verification missed.
Re:Jepsen: Datomic Pro 1.0.7075 by nine_k on Thursday May 16, 2024 @02:22PM (#40381525)
What kind of flaws? I would expect performance problems.
Re:Jepsen: Datomic Pro 1.0.7075 by jupp0r on Thursday May 16, 2024 @04:37PM (#40382813)
To start with, you usually perform verification on a behavioral model and not on the code itself. This opens up the possibility that there are behavioral differences between the code itself and the model which wouldn't be caught.
Re:Jepsen: Datomic Pro 1.0.7075 by nine_k on Thursday May 16, 2024 @05:01PM (#40383082)
Could you offer an example?
Re:Jepsen: Datomic Pro 1.0.7075 by jupp0r on Friday May 17, 2024 @01:42PM (#40392393)
If you work through the TLA+ tutorial[1] it will help you get a good idea of the benefits and limitations of verification.

[1] https://learntla.com/

Re:Jepsen: Datomic Pro 1.0.7075 by asa400 on Thursday May 16, 2024 @02:36PM (#40381688)
Is there anything I can read about what capabilities Datomic requires of the underlying storages it uses?