Re:1 in 4 Statisticians Say They Were Asked to Commit Scient...
by geezerjay on Saturday November 03, 2018 @08:58AM
(#18369973)
> They call bugs 'details' which, I find, is a frightening state of mind for someone publishing an algorithm.
It really depends on the algorithm. For example, a bug in the random number generator of a stochastic search algorithm that affects, say, the variance of a distribution won't have a relevant impact on the outcome.
Re:1 in 4 Statisticians Say They Were Asked to Commit Scient...
by mlthoughts2018 on Saturday November 03, 2018 @10:32AM
(#18370325)
That could have a huge impact. For example if it affected random draws of a hyperparameter in a Bayesian model, leading to incorrect credible intervals ultimately in the posterior distribution. Or worse, if the RNG bug was affecting the variance of randon samples in some deep component of an MCMC algorithm like NUTS or even simple Metropolis. Depending on the exact nature of the bug, it could even cause the model to violate Detailed Balance, and entirely invalidate sampling based inferences or conclusions.
Re:1 in 4 Statisticians Say They Were Asked to Commit Scient...
by riskneutral on Saturday November 03, 2018 @12:38PM
(#18371080)
Well that is why you would always run statistical validity tests on the RNG and other intermediate values whenever using a Monte Carlo model in production. Ideally the tests should run as part of a Continuous Integration workflow.
Re:1 in 4 Statisticians Say They Were Asked to Commit Scient...
by mlthoughts2018 on Saturday November 03, 2018 @02:35PM
(#18371742)
If you (dubiously) write your own RNG, sure. But you should never do that. And for library RNGs, you should execute the unit tests of the library. Frankly, running it as part of CI is at best overkill and at worst adds complexity that costs you. If you pin versions of your dependency and isolate the artifact into an in-house artifact repository, so that the library code is literally never changing unless you explicitly modify the version, then you should test it up front, then again only occasionally if you actually have evidence of a bug. And as part of any code review of the code change that introduces a version change.
> We also noticed significant improvements in performance of RND every time we discovered and fixed a bug [...]. Getting such details right was a significant part of achieving high performance even with algorithms conceptually similar to prior work.
They call bugs 'details' which, I find, is a frightening state of mind for someone publishing an algorithm.