News for hackers. Stuff that matters.
Re:A cloak-and-dagger tale behind an anticipated result in p... by shmageggy on Monday February 01, 2021 @12:52AM (#25985961)
Really impressive effort to avoid the garden of forking paths. I wonder if (hope that) this sort of thing will become more standard in research.
Re:A cloak-and-dagger tale behind an anticipated result in p... by exmadscientist on Monday February 01, 2021 @02:43AM (#25986496)
It has been standard for a long time in particle physics. The experiments that I worked on were blinded in similar ways, though a bit less elaborately. On my would-be thesis experiment all the blinding turned out to be pointless... the signal was as bright as day. Always good when that happens!
Re:A cloak-and-dagger tale behind an anticipated result in p... by dash2 on Monday February 01, 2021 @07:35AM (#25988083)
I've been working with an econometrician, what really impresses me is her insistence that we don't do any analyses until we've figured out what the right ones to do are. No exploring because it will taint your p values! It's a hard discipline to follow.
Re:A cloak-and-dagger tale behind an anticipated result in p... by PartiallyTyped on Monday February 01, 2021 @08:03AM (#25988254)
Makes sense really. From ML; the test data are an unbiased estimate of the performance of a model and you can use it get PAC (probably approximately correct) estimations of your performance. There are three (4) powers at play here, the confidence [1], the error bound [2], and the number of data points (and model complexity [3] but that is not relevant when we compare a single model against test data). Your performance is exponential in the number of samples you have (meaning more data gives you exponentially better bounds and more confidence), alternatively, for a fixed sample, you can have either high confidence, but weak bound, or small bound but low confidence. When we consider the training data as an estimate of the expected performance, we also take into consideration the class complexity.

The moment you touch the test data you have tainted it because your brain can do a crude search on the space of solutions and reconcile them after the fact. This is why any preprocessing and cleaning is parameterized on the training set and then applied to the data, that also includes the class of models you can consider and is why you never select final models based on their performance on the test set; it becomes biased.

[1] probability of getting a good model.

[2] how much the empirical performance underestimates the expected performance.

[3] the number of models you can select from a particular class of models.

Re:A cloak-and-dagger tale behind an anticipated result in p... by Gatsky on Monday February 01, 2021 @03:40AM (#25986764)
I think this is possible because of the exclusivity of the equipment. These scientists have the luxury of being the only ones able to conduct this experiment, and presumably some long run funding to do it. If there were 5 different teams competing, there would be less rigour I suspect, since other metrics (being first, number of papers, the usual academic swill) would come into play.
Re:A cloak-and-dagger tale behind an anticipated result in p... by mhh__ on Monday February 01, 2021 @03:59AM (#25986893)
I applied to do an internship working on air to air missiles for exactly this reason.

Unluckily for them their online literacy says I'm too stupid to write simulation code them so their loss.

But imagine the FPGA-grunt in those things!