In certain cases, you know what the neural network should do, for certain inputs, and you have a quite clear idea how each component of the network would solve this, and this should be doable, and with a bit of work you could also construct the parameters by hand, such that it works at least for non-noisy constructed toy input data.
Actually, I think for more complex tasks, having such intuition would anyway be a good idea.
Now, you could use a SAT solver such that it does the work mostly for you. You formulate some constructed inputs/outputs, maybe some other constraints, and let it solve for the parameters. This would be a good parameter starting point for real world data. And if the SAT solver fails to find any solution, maybe your neural network is actually not powerful enough.
- The Pragmatic Programmer -
This changed my life when I started programming 20 years ago. Most of the practices are now common, but it was almost radical back then.
- Designing Data Intensive Applications
This is so well written, so elegantly fundamental. It's an absolute pleasure, even if I don't really refer to it in practice.
- Structure and Interpretation of Computer Programs
In many ways, the MIT version of PAIP. Its elegance complements the pragmatism of PAIP well.
- The unicorn project
My number 5 book varies often, because I haven't found many books that reach the writing quality of the other 4. This is a business novel, but it really solidified a lot of concepts for me: lean, queues, theory of constraints, what devops is about, working as a programmer in a business.
* Concepts, Techniques and Models of Computer Programming: https://www.info.ucl.ac.be/~pvr/book.html
* The Art of Prolog: https://www.dropbox.com/s/2umr9ouz0jdelio/1407.pdf?dl=1
* Probabilistic Models of Cognition: https://probmods.org
Something that merges logic, types and proofs, perhaps yet to be written. Some good preliminary material:
* Program = Proof: https://www.lix.polytechnique.fr/Labo/Samuel.Mimram/teaching...
* Concrete Semantics: http://concrete-semantics.org
* The Hitchhiker’s Guide to Logical Verification: https://raw.githubusercontent.com/blanchette/logical_verific...
* Programming Language Foundations in Agda: https://plfa.github.io
* Logic and Computation Intertwined: https://cs.uwaterloo.ca/~plragde/flaneries/LACI/
* Software Foundations: https://softwarefoundations.cis.upenn.edu/
Prolog is one of those things that has alluded me all this time. Mostly I don't think I've had an application for it, and, bluntly, at least for me, I need to have a "real" application to solve to best learn something. Seeing the "animal" program repeated over and over and over again was never any help.
In hindsight, maybe it would have been appropriate in an email messaging application I did long ago. It's message routing workflow was not inscrutable, but certainly difficult (and it didn't help that the route could split, sending the message to more than one place with their own workflows -- that was fun).
I've done a bunch with rule engines (and the message routing was done with an ad hoc one), but less so with inferencing.
Maybe this book will give me some insight to explore further. It's always one of those things that sort of nags the back of my brain that I don't quite grok it.
The Craft of Prolog is also very much worth looking into.
A related approach is ASP, which combines SAT with logic programming. These are the two canonical books:
* Answer Set Programming: https://www.cs.utexas.edu/users/vl/teaching/378/ASP.pdf
* Answer Set Solving in Practice: https://potassco.org/book
A great opportunity for the reader to expand them.
I've seen for example the Scheme implementation from PAIP expanded and integrated into a CMS.
Old AI is today's bleeding edge computer engineering. There is an enourmous amount of free lunches for computer engineers and software startups in the old school artificial intelligence.
* modern SAT solver performance is impressive. They can solve huge problems.
* Writing a complex systems configurator with Prolog or Datalog can be like magic.
* Expert systems. There has never been so much use for them than today. Whenever you see expensive systems utilizing complex mess of "business logic" and expensive consultants, you should know there is a better way.
(I use SAT-solvers to partially initialize neural network parameters).