I think of neural networks as "smooth compression" - something that compresses a key-value database into a continuous non-linear function. Previously Daniel Holden wrote about something similar in his blog (https://theorangeduck.com/page/machine-learning-kolmogorov-c...)

I agree with the author's sentiment that "learning" might be a misleading term. Neural networks can just be seen as specialized programs that just compress certain kinds of data extremely well (at the cost of extremely high pre-computation). But it's still a very useful technique to tackling previously intractable problems.