Plus there are reasons to treat functions of functions differently. Often there are subtle assumptions made on the domain of a function of a function. The space of functions is just too damn big to work with naturally. Almost none of the things you want to do to functions are applicable to all functions. R^R consists mostly of pathologies and monsters.
That the set theoretic definition of "function" as a functional relation between sets is rarely useful isn't really so surprising that you need to emphasize that you use a more reasonable notion of function all the time.
And the integral transform notation is frequently changed, but only ever locally and in different ways by different authors. Pick up two books on "Fourier transforms for engineers" and I promise you that you will find different notations for the same thing and probably even different notations within the same book. And none of them will be good.
Maybe I'm missing something, but I don't see an issue at all. Perhaps you can present some examples?
I understand where you're going with the Fourier example, you often need surrounding context to know what's going on, you need to know what it is used for.
This is perhaps the biggest difference between math and computer notation, but it's a feature, not a limitation. If you want to you could easily repeat all the variables on the left side of each expression, to make it clear what the frequency domain variable is. But you don't do that, because there is no need for each expression to stand on its own. It's not even needed in all computer languages, e.g. Swift is typesafe but type is inferred and often not even explicit anywhere.
The latter is crazy. Consider the expression "F{e^{-t^2-y^2}}", which might stand for the fourier transform of a 2d gaussian, or it might be the fourier transform of t, with a parameter y, or it might just be the fourier transform of a constant function, or... It is only really defined in the surrounding text. The notation is incomplete and while in this example that might not be such a large problem it just gets worse as you pile on complexity. All integral transforms are written like this, as are expected value, variance and so on.
A lot of times notation in math is choosen to be suggestive. For example, the integral and sum notations are actually pretty neat, since they make common manipulations more visually clear. In particular, since the order of integration doesn't matter, putting the binder inside the integral as a "factor" is actually pretty inspired - commutativity makes Fubini obvious. The "d/dx" the author complains about can be made precise and is similarly great. Consider "dx/dz = dx/dy dy/dz", doesn't this just look entirely natural? But there's something about manipulating functions as first class objects that seems so unnatural to mathematicians that it needs cryptic notation to ward of the unwary...