So, in other words, software engineering is lacking a whole lot of engineering?
I mean, there is copious engineering effort internal to, e.g. the Linux kernel.
And much goes into RFCs.
But is there much internet-wide engineering going on to squeeze out the entropy?
Repackaged: the engineering that occurs is more bottom-up than top-down. Google, for example, has enough heft to say: "QUIC: here is a bit of top-down engineering." But that example bolsters my general point, in my opinion.
Is there a hidden assumption in "unnecessary"? None of the tech unfolded in a deliberate, methodical manner. It was random people doing various things over time and a plethora of details.
To assume that the aggregate would have a tidy structure seems unrealistic.