Re:No, DeepMind has not solved protein folding
by dnautics on Friday December 04, 2020 @03:40PM
(#25307596)
> for example in NLP, many people assume a high log prob score means a high confidence but it is not true at all
Thank you for the clarification. I wasn't aware of this, as I'm most familiar with super-basic/old NLP techniques like BOW/RNN/LSTM/GRU techniques, where log prob scores seemed to me to be roughly correlated with result quality. I'm aware the landscape has changed recently with insights about high dimensional searches...
Re:No, DeepMind has not solved protein folding
by visarga on Saturday December 05, 2020 @02:47AM
(#25312690)
There is no difference in the log prob between LSTM and transformers. It just depends on the number of updates you make (batch size, epochs), if you train it to overfit, then your log probs will be pushed closer to 1 or 0.
But after training you can recalibrate the temperature of the softmax on the test set and still get meaningful confidence scores (temperature calibration). Or you can use a variation of cross entropy called Focal Loss that will leave your logits un-squashed.
Right but we also can't generalize from 2 out 3, and without knowing this figure it's really hard to say how useful this is, no?
> Knowing how ML works, I'm surprised it didn't give any indication of low confidence.
This is actually a hard problem in ML, for example in NLP, many people assume a high log prob score means a high confidence but it is not true at all.