Part of the issue is that many of the neurons in a neural network end up contributing little to the performance. This is why the practice of pruning, the removal of neurons that don't contribute much, exists [1]. Neurons can end up having so little gradient that they end up stuck (Big problem with RELU), or can end up being insignificant due to the subsequent layers weights of that neuron.

As the author is using RELU, he will have a decent number of neurons 'die'. So some 'over-provisioning' is not a bad idea in theory. Also, if my keras isn't so rusty, I think the author is using less parameters than you are stating.

Still a more reasonable dropout rate and maybe some regularization/batch normalization might help, but I would say not over fitting on only 700 samples is a hard task, even with a network much smaller than that.

I would be pretty shocked if this neural net wasn't over fit

[1]-https://towardsdatascience.com/pruning-deep-neural-network-5...

I know this is a joke, but the theory of generalization in NNs is rapidly advancing and it's not quite that simplistic: https://arxiv.org/abs/2003.02139

Ya. And the choice of optimizer (in this case adam) also imposes upon it some regularization scheme.

I just thought I'd highlight a bit of funniness.

How does Adam provide regularisation? I’d never heard of this before and I don’t recall it from when I read the paper.

Just use dropout, dropout prevents overfitting!

/s

/s for this post, I mean. I have had this very suggestion made to me non-sarcastically under similar circumstances.

Yeah, that's what I thought too. Interesting work nonetheless!

what a meme haha :)

after many similar comments I plan on implementing a simpler model and seeing how it compares

I forked your colab. There was abit of code it couldn't run. `Gxxx_matrix.csv` not found.