With me, it had to do with the languages I wrote (C and Perl), that required CGI to publish. CGI did fine, but it's fairly primitive. Just another system text pipe (which, TBH, is pretty much what is going on, under a Webserver).
But for most folks, CGI is unapproachable. That's what PHP gives people. It was so easy to use, that anyone that could edit an HTML file, could also write powerful programs in PHP. It was integrated into the Webserver, not the system (as far as the user was concerned).
At the time I first encountered PHP, ColdFusion was an industry standard for that kind of thing, but it was a commercial product, locked to a single vendor. PHP gave hosting companies a fairly simple package (for free) they could compile into their Webserver, that would give their customers that power.
Nerds have no issue, using things like CGI, but most folks aren't nerds, and we sometimes have a real hard time, understanding that. Nerds that do, often become rich.
So I was literally trying to know more. (And I understand a bit better now, thanks.)
I think that anything that will interact with remote browsers, should go through a Webserver. Many utilities actually have their own built-in webservers (not sure that’s always a good thing).
Because the simplicity results in various footguns, just like most programmers probably shouldn't use C these days ?
("Webserver" sounds like an extra layer on top, like how Python allows you to (mostly) forget about memory handling issues ?)
Is it also because simplicity either doesn't add performance here, or is it just because the performance is rarely needed ?
It’s what systems programmers would like, because we don’t like things that get in the way. However, most folks aren’t systems programmers. They want padding and safety nets.
I mentioned writing CMSes, and one of them was in Perl (using CGI). I still wake screaming, sometimes, and that was probably 25 years ago.