The whole story and ideas put into Windows installing programs are a stupid joke. It's designed by project managers who have no idea what they are doing and no imagination and is executed by the cheapest least motivated programmers from South Asia Microsoft can possibly find.
A lot of people who do useful stuff for Windows try to stay away from as much of Microsoft's stuff as possible, and integrate it as little as possible with anything else coming from the system because it's just maddening how horrible everything there is.
"Plain-Text files for configuration? what do you think we are? savages? no, we need a hierarchical registry! every serious program must use some opaque binary format to store stuff!" seem to be the general animus at that time. Nowadays, even if you really hated the idea of a text files for configuration in your home direction, people would probably do something more straight-forward like using a SQLite db.
There were also very practical solutions incorporated to accommodate the constraints of the hardware of the time that come off looking like needless complexity today, too. (There's also, arguably, some laziness in the old binary file formats that were nothing more than dumps of in-memory structures. That's a common story across a ton of old software-- not just MSFT.)
Rob Mensching, who worked at Microsoft on Windows Installer pre-release, has a nice blog post about internals of MSI.[0] He goes into some of the overwrought architecture in MSI, as well as quirks to overcome performance limitations in a world of floppy disk-based distribution and much smaller memory capacities. It's a good read.
[0] https://robmensching.com/blog/posts/2003/11/25/inside-the-ms...
Yes, people preffer to debug their own code rather than spend shitload of time to understand Wix/MSI.
Microsoft deciding early on to not produce low cost tools for Windows Installer also didn't helped with the adoption.
But there is the modern implementation of AppX Bundles which was later extended to create MSIX which allows app distribution without the windows store. There are still drawbacks to using MSIX usually because you want to touch Windows in ways you can't inside the sandbox.
In my experience, you can pretty easily write the nice sandboxed MSIX manifests by hand, it's not too bad, but general MSIX doing weird MSI things you still want better more expensive tools to build them (and of course Microsoft themselves still don't exactly provide that and will point you to plenty of expensive third party installer studios for options, many of which are the exact same ones people have been overpaying for decades).
[0] The one complaint I'm aware of is that you can't do custom installer UI and "attended" installs with user choices. There's one MSIX UI and it is barebones but acceptable. That's all you get.
Often, as you mentioned, Windows Installer packages are wrapped by an executable (in WiX this is called a "bundle" because you may also choose to add redistributables like the C++ runtime).
However, what you see in installations like SQL Server, Office and Visual Studio is that the installers are bundles as well - of a large amount of MSIs that need to be installed and uninstalled in a specific order. A single Microsoft Installer package is transactional and can be rolled back on failure, but bundles are not as well defined and left open to the implementation of the bundle. Windows Installer does not reach beyond the single msi package.
It is intentionally obtuse at times (MSIFileHash table uses rearranged MD5 hashes for example), and also many features made sense for the 90's era where bandwidth was low and connectivity limited, and lots of stuff was distributed on CD's. The look on people's faces when you explain advertisement to them the first time... How their unrelated app can get stuck in a loop of repair for a piece of unrelated software...
It was deprecated by the newer AppX/MSIx/AppV format which uses sandboxes, binary chunks/streaming and no executable code to install stuff.
For my own desktop computing, I prefer MSI packages because I prefer having control post-install to tinker with things if I feel like it. Also, I have the skillset to modify the installer to my whims if I so choose.
I can offer a little perspective on MSIX, having devoted months of my life to it in a past job.
MSIX is nearly unusable outside the Store. It will work in a tightly controlled environment, but when you try to deploy it to a wide variety of users you will run into 1) unhelpful errors that basically can't be diagnosed, 2) enterprise environments that cannot/will not allow MSIX installs. I get the impression that the MSIX team is uninterested in solving either of those issues.
It's not a coincidence that virtually no first-party teams use MSIX to install their product outside the Store. Office tried for a while and eventually gave up.
Despite all that, there are still a few people at MS who will tell you that MSIX is the future. I don't really understand it and I assume there's a weird political reason for this.
Still, when you finally make it work you get a lot of benefits. MSI is clearly dead end tech which is why so many MSIs are just installer EXEs wrapped in the file format. It doesn't have any clear path to modern essentials like online updates, keeping the OS clean, sandboxing and so on. If you were on the Windows team, what would you say the future was?
For enterprise environments it's actually somewhat the opposite: MSIX packages can be installed without admin privileges due to their declarative nature, and it's very easy for admins to provision MSIXs to Active Directory groups because they don't have to do any repackaging work. Yes, some admins have hacked Windows to stop it working because when MS launched the Store they didn't provide any way for admins to opt out, but these days they have the knobs they need. Also, because they're just zips you can always just unzip them into your home directory to get at the app inside. It won't auto update that way, but as long as EXEs can run from the home dir it can work.
Products like Office and Visual Studio have entire teams devoted to nothing but their installers, which is clearly going too far in the opposite direction. Most products won't want to do that.
I assume the Here in NIH refers to an individual team, not MS as a whole.
Teams is entirely NIH https://github.com/Squirrel/Squirrel.Windows for updates to the Electron app.
I would use winget, but MS made it weirdly hard to run as a script on multiple computers, it installs per user, because... who knows.
So still using chocolatey
Therefore the documentation is poor , like the absolute worst I've ever seen. Opening issues for doc issues never results in anything. Pointing out UX issues is usually shot down. Finally, until this year you needed .net 2 installed to build it, which does not play well with windows docker.
I get unattended installs and uninstalls "for free" when well-behaved applications use Windows Installer. Patching is included, too. Customizing installations is fairly straightforward.
On the developer side it has historically used a very quirky proprietary file format (MSI) with a fairly steep learning curve and a ton of "tribal knowledge" required to make it work for all but the most trivial cases. (Though, to be fair, most installs are the trivial case-- copy some files, throw some stuff into the registry, make some shortcuts.)
Worse, it allows for arbitrary code execution ("Custom Actions"), at which point all bets are off re: unattended installs, removal, etc. Some Windows Installer packages are just "wrapped" EXEs (Google Chrome, for example).
I've packaged a ton of 3rd party software as Windows Installer packages. It's an ugly system with lots of warts and legacy crap, but if you need to load an application on a large number of Windows PCs reliably unattended it's decently fit for purpose.
There is reasonable free and libre tooling to generate MSI packages from plain text source (the WiX toolkit) and it can be used in a CI pipeline.