My argument: the theoretical limitations of NNs (lack of modularity, symbolic reasoning, verifiability) cause no practical problems to usefulness - we can just analyze the code artifacts as we do with human programmers. Do you disagree?

Yes. These limitations are not theoretical at all. The author touches on compositionality --- how a problem/program be decomposed into smaller, orthogonal problems/programs, reasoned about and tested separately, and then abstracted away in an interface that hides the implementation details. This is the essence of programming and software engineering at large, whether you're programming in assembly, Java, or Haskell. To divide and conquer so that we can fit an isolated aspect of the program in brain cache so that we can reason about it. This is a fundamental limitation and will not change until the year 40,000 when we have Space Marines.

A neural network, conversely, is a big ball of mud. Impossible to reason about and to test except for whole-system, end-to-end testing, which is impossible to do exhaustively because of the size of the state space. It is, by design, unexplainable and untestable, and therefore unreliable. It's why you use globals in C only judiciously. (I am just rephrasing the article here, not saying anything new.)

And the evidence that it causes practical problems to usefulness is already out there; "hallucinations" are simply errors, just that corporate PR likes to pretend that it's a "feature" and not a bug. This is delusional. A society seeking digitalization should run away from this level of stupidity.