I think you can illustrate the difference very well with basic algorithms: We absolutely do know how sorting algorithms "really work", there is basically a traditional first-year CS lecture that is all about that. With "really work", I mean here that we know which principles make sure that the algorithm always returns the result that we expect - a sorted array - even for for unknown inputs. You can mathematically prove that a sorting algorithm will return the correct result for every single input out of a literally infinite set of possible inputs.

Part of the definition of the algorithm and of the proof also involves creating "intermediate concepts" that capture some sort of structure inside the algorithm: If you just measured all the electrical charges inside the CPU, you wouldn't see much: just a bunch of memory cells changing state in seemingly "unpredictable" patterns until at some point, "magically" the result appears.

However, with sorting algorithms, we know which memory cells represent arrays, the instruction pointer, pivot elements, etc. We know there is a specific way those cells are supposed to interact and we know why those interactions will in the end lead to a fully sorted array.

The CPU itself is a similar example: It's trillions of transistors, switching in nanoseconds - but we can still explain what each transistor "does", because we know the higher-level functional groups that they belong to - such as logic gates, then counters, then arithmetic units or memory, etc etc. Conversely, if there is an error, we know how to trace it back.

I feel with LLMs, we're still very much at the "measure the elextrical charges" stage: We can pass words to the network that resemble an instruction. The words are converted to a vector, which is transformed through a number of very large weight matrices and in the end is turned into a probability distribution on words - and if we sample from that distribution, we get words back that very much look like the execution of the instruction.

However, that doesn't itself explain what principles result in the network reliably mapping a human-readable instruction to its execution. It doesn't tell you about higher-level functional units within the weights. That's what I mean with "understanding".