I am absolutely certain it is possible to explain this stuff without using jargon and mathematical notation that is impenetrable to the majority of professional software engineers.
I agree, but that doesn't mean that every article on the subject needs to be written for that audience in that language.
At some point, at some level, you really do need to just learn what a dot product is and what a matrix is. It's not weird notation or jargon, these are fundamental concepts.
Just like if you want to really learn how programs work you can't refuse any explanation that talks about "variables" or "functions" because that's jargon.
You can explain it, but it's going to be more at the level of "the network looks at the words" type explanation.
The idea of the transformer somehow being a trainable key-value store is kind of abstract and weird and has little to do with the mathematics of it. The math part of that is how the dot product encodes for similarity between vectors, but beyond that it really is a "if you get it you get it" kind of thing.
I agree that there probably could be a better "on ramp" into this material than "take an undergraduate linear algebra course", but ultimately it is a mathematical model and you're going to have to deal with the math at some point if you want to actually understand what's going on. Linear algebra and calculus are entry-level table stakes for understanding how machine learning works, and there's really no way around that.