I can relate to your point more in that way. Joel has a gift for exlplaining intangibles. Agree with his point also having had to spend time debating whether an abstraction choice adds value.

Still worth noting I think though that he does say “sometimes”, because no one is going anywhere without abstractions or generalizations. The problem is they’re hard to do really well and we know historically really smart people can make small mistakes and fall into the not cogent/no value/Joel trap #234.

That seems to be a poor analogy to relate in this case. I’m not going to claim that the parent article is factual or even useful but at first glance it seems interesting.

In general the difference between a tinkerer and an engineer or scientist is that the latter uses appropriate mathematical models to garner deeper insight into problems than available to general tinkerers and "smart thinkers". If the linked article succeeds in linking any of the concepts of thermodynamics — which is one of the core building blocks of modern physics and relevant to almost every field of practical engineering — to DL models then that’d be huge boon. There are a lot of well honed techniques and tools available in the thermodynamics and statistical physics toolboxes which could vastly improve the ability of DL to be tuned for useful tasks in a scientific manner. Most application of DL appears to be "dark arts" or alchemy phase where only intuition is used to direct what problems can and should be tackeled using DL. It seems every other paper published in DL is just happenstance.

Joel Spolsky‘s comment really fits more with smart "tinkerers" but not serious attempts at modeling of the energy minimization aspects of restricted Boltzmann machines (ahem DL) which really do borrow a lot from thermodynamics [0]. I’m glad "smart thinkers" like Geoffrey Hintonmidday know when to stop [1], and kept generalizing and tweaking Boltzmann machines. Heck even the idea of applying energy minimization techniques from the phase changes of spin glasses is one of those applications of abstract high level models that really don’t mean anything. Except when evenetually they provide entirely new insights and, well fields of study. Of course there’s a difference between serious attempts and hard work of establishing mathematical and conceptual correspondents between fields and just quackery or pseudo-sophistication. Given the amount of formula development in the linked paper I’ll take it as a good faith attempt to seriously link fields.

0: https://en.m.wikipedia.org/wiki/Boltzmann_machine#History —- especially the first paper on spin glasses 1: https://medium.com/@andreykurenkov/a-brief-history-of-neural...

> “If the linked article succeeds in linking any of the concepts of thermodynamics — which is one of the core building blocks of modern physics and relevant to almost every field of practical engineering — to DL models then that’d be huge boon. There are a lot of well honed techniques and tools available in the thermodynamics and statistical physics toolboxes which could vastly improve the ability of DL to be tuned for useful tasks in a scientific manner. ”

I think this is just wrong. Firstly, statistics has already been linked with thermodynamics for a long, long time, and people already have been using ideas about thermodynamic laws to describe potential functions and algorithms like Hybrid Monte Carlo and simulated annealing, even borrowing convergence metrics for things like parallel tempering.

When that connection is practical, it is about explicit, case by case algorithms, and not at all about recasting the generic interpretation of models into some other thermodynamic framework, which is just a type of cute toy analysis.

I would say specifically the example of Boltzmann machines highlights this exact point between a result that has pragmatic implications and a result that is just playing around with definitions.