Feynman had a funny story about this. I’m no Feynman, but he invented new ways of writing sin, cos, etc. He said he disliked the way it looked, since cos(x) looks like cos multiplied by x. And of course the story ended with the same punchline you outlined: when you want to talk to others, you need shared vocabulary.

But the thing is, it’s extremely easy to remember to say “loss” instead of “penalty” when I’m talking to someone. But it was extremely hard for me to even understand what the heck a loss was. What is it, exactly? What’s it doing and why? How should I think about it — and more importantly, how can I extrapolate that thinking to take advantage of it?

Maybe it’s a personal quirk, but I simply couldn’t understand loss. I know penalty though. Ditto tor learning rate vs step size. So it’s more of “internal advice” rather than me saying that you should rewrite your papers with the new names.

EDIT: By the way, I wasn't proposing that "penalty be renamed to "regularization". I was under the impression that what the parent comment was calling penalty" was normally called "regularization", i.e. that regularization was the formal name for it. If that's not true, it's possible my understanding is incomplete -- what is penalty? I haven't heard of it till now, to be honest. And googling for "machine learning penalty" pops up 5 articles on regularization.

So I was proposing two changes: loss -> penalty, and learning rate -> step size.

I guess I'll call loss "punishment." It matches how it feels to make progress in ML anyway.

I have looked into the historical context and reason for the use of the word (the technique was first popularized in something which "regressed to the mean"), as well as its development, and it still bugs me any time.