> [Self-plagiarism] doesn't mean "I use the same turn of phrase every once in a while". It means "I reproduced substantial portions of prior research or other text of my own and passed it off as new research."
The term is not used to mean occasional repetition of phrases or concepts, nor does it apply when that expectation is not the production of new research (e.g. when delivering a course lecture). The term is used to mean something more along the lines of when you take a paper you published some time ago, make a few edits, and re-submit it to another journal.
I once reviewed a paper. It seemed familiar. I pulled up a paper by the same authors from a decade previous. Yup, the first 10 pages were almost identical. I had read the earlier paper, and others had cited it -- though not the authors.
That was a waste of time, adding nothing to the dialog.
As for who the victim is, typically the direct victim would be the funder: if you received a grant to pursue novel research and it turns out you were just rehashing prior work, then you have stolen funding. Similarly, if you are under the employ of a university, then you have stolen funds by reproducing old research and passing it off as new...instead of actually producing new research. These are of course only the monetary victims, the broader victims are those robbed of scholarly advancement by having your job title filled by someone who recycles material versus advancing the field.
I see no reason to change terms and structure in scientific publications so as not to plagiarize myself.
There generally is no reason to change terms and the like. The solution is simply to cite one's prior work when quoting from it - the lack of the latter (citing that a given text is one's prior work) is what constitutes self-plagiarism, not the mere reuse of terminology.
While it is generally against most codes of conduct to plagiarize your own work (for example, publishing something you published 10 years ago again, without a notice that it has been published before), I have never seen any cases of anyone actually being penalized for doing so, short of issuing a new edition with a citation to the prior work.