One reason we will see more deep learning in electrophysiology is simply because it allows one to abstract over a good deal of statistics and signal processing, which are hard. Deep learning is being used as a black box for those who can't or don't want to understand their data.
It’s pretty insane to think that a decent neural net trained on sufficient data wouldn’t be able to outperform basic regression.
Well, maybe it's logically obvious, but it's also empirically false: https://www.youtube.com/watch?v=x7psGHgatGM (Skip to 13:36 to get to the point)
> But a neural network can model non-linear relationships as well. Now ask yourself, what is the likelihood of there being a non-linear relationship in the data? As with most real-world data, the likelihood of non-linear relationships is extremely high. That’s why we can be quite certain a neural network approach will able to outperform linear regression.
As I've said elsewhere in this thread, a deep neural network of N layers can be decomposed into a complex nonlinear function of N - 1 layers followed by a linear combination of the nonlinear features generated that function. There's no law that says that you have to use a neural network to generate your nonlinear features. You can use any method you like and then linearly combine those.
According to who? Did you watch the talk I linked? Because in that video, NeurIPS gave their test of time award to a paper where they generated nonlinear features and used those to train a linear classifier. I guess the committee at NeurIPS also didn't understand what they are saying?
> And even if they were, a neural network will do a much better job of generating the non-linear features than you will be hand. So again, it’s logically obvious that a neural network will be capable of performing better than simple linear regression.
Again, logically obvious but often empirically false.
Regression models can't capture that. Now it's possible that the behavioral nonlinearities are somehow emergent, and that despite those, the ECG can be captured by a linear model. But I wouldn't want to bet on that.
Question: are you saying the final layer is "in effect" a linear combination? At least in transformer architectures, the dense end block is iirc three layers deep and uses relu. Even if the CNN's dense final part is one layer deep, wouldn't it also be using relu activations?
Even a one layer dense ANN can capture nonlinearities if it has a nonlinear activation fn. But maybe in practice the activations don't do a lot of work? Or am I simply mistaken about final layer, does it simply have linear activations?
Also, can you share your intuition about the nonlinear features? Spectral/wavelet analysis? Or something more complex?
Not just in effect a linear combination; it is a linear combination. There are some exotic nonlinear NN layers that are used in particular niches. But in general, NN layers are syntax sugar over matrix multiplications (i.e. linear functions). The nonlinearities are the activation functions between the layers.
> But maybe in practice the activations don't do a lot of work?
No, the activations do do a lot of work. There are real nonlinearities in the network.
> Or am I simply mistaken about final layer, does it simply have linear activations?
Sometimes you actually do use a linear activation on the final layer for certain regression tasks, but that's not the main thing I'm getting at.
Let's say that your final layer has is a ReLU activation. What is this conceptually? You are taking a linear combination of the features from the previous layer and them clamping the result to >= 0. Sure, that's a nonlinearity, but it isn't going to have much in the way of emergent modeling capabilities. You need to stack many, many nonlinearities before you get that.
So my point is that a deep neural net of N layers boils down to a complex nonlinear function of N - 1 layers, followed by a "dumb" linear combination in the final layer. You can do this with traditional ML methods as well, but you have to handcraft your nonlinearities.
> Also, can you share your intuition about the nonlinear features? Spectral/wavelet analysis? Or something more complex?
There's innumerable possibilities here. It could be starting with a method that's inherently nonlinear, like nonlinear PCA, polynomial regression. Or it can involve transforming the output of a linear function (like the Fourier transform) in a nonlinear way.
Admittedly this very tough. And for really tough problems, like video synthesis, effectively impossible. But NNs get thrown at much simpler problems all the time.
I spent half a decade on a data science team developing algorithms for ambulatory ECG monitoring at scale. 100+ human years of 256 Hz multichannel ECG collected everyday. Waves, beats, & rhythms all segmented and classified. PDF reports generated for cardiologists and EPs.
DL techniques like unsupervised pre-training blew away every other classical or purely supervised method we ever tried. Especially for rare and critical arrhythmias like VF. It unlocked value in massive unlabeled datasets. It gave us a framework for scaling model performance as a function of dataset size and compute, with no fundamental limit.
I do get your frustration about the publication game. You might find industry is more aligned with pragmatism & efficiency.
Apart from the total lack of external validation (which I suspect will never come), we already know that ECGs have features associated with emotion: heart rate, frequency of premature ventricular contractions, frequency of premature atrial contractions. Are these deep learning detectors better than running a simple regression after using something like a wavelet transform to detect these specific features? This matters because if you have a method that is more computationally efficient, more easily interpreted, and more robust to extrapolation to out-of-domain data, why in the world would you go with DL?
Of course the article is completely silent on these questions. If I was a reviewer I would refuse to accept until they added extra experiments to address this.
This is a symptom of a more general problem in interdisciplinary medical research: Papers are written and reviewed either by engineers who have limited understanding of the clinical background or by clinicians who can code a CNN in pytorch but have no knowledge of classic ML methods or ML theory. And so we see seriously flawed papers get accepted. (I know this is a preprint, but I've seen similar stuff in well regarded journals.)
It's incredibly frustrating as someone who tries to do truly rigorous research in this space, only to be desk rejected because ECG magic sounds a lot cooler than incremental gains.