I remembering hearing at the time that this was about hardware support and/or app compatibility. But my cynical side just thinks they wanted to justify charging more for a workstation SKU for another release cycle.
Microsoft was planning on building an entirely new Internet-focused consumer version of NT. It was too ambitious for its time, so to meet deadlines, most of it was scrapped and some UI elements were backported into an updated Windows 98 which became ME. This is why Explorer for both ME and 2000 look similar despite having little else in common.
Other elements of the scrapped project made its way into XP, but barely compared to what it was supposed to be. I think XP’s fancy login screen was one of them.
If anyone’s interested I’ll try to find where I read all of this. It might’ve been on Paul Thurrott’s Windows Supersite or another site that collected Windows betas.
It’s interesting to see twenty year old experiments with flat UI designs considering how dominant that style is today with Android 5, iOS 7, Windows 8, etc.
And now many apps use HTML/JS/CSS for styling and rendering.
I think software compat was also lacking. DOS games were still in people's libraries (eg Quake from June '96), and I remember needing an extra 4MB to run those inside Win9x versus booting to DOS mode. That stuff didn't Just Work (tm) on Win2K. For example, http://sandmann.dotster.com/djgpp/DJGPP_W2K.htm is a list of issues with the most popular GCC port to DOS. (The Quake DOS binary was actually built with an earlier version of that compiler)
I remember the WinXP DOS emulation was supposed to be much improved over Win2K, but don't recall specifics.
There should be some rule internally at Microsoft that institutes a complete ban on new settings UI unless it actually replaces old UI. The amount of times I need to fall back to the old UI is absurd. Also - I'm not sure programmers appreciate how jarring it even is to have the same settings presented in two places. It forces users to construct some kind of mental model of the behind-the-scenes-actual settings, because of course the two dialogs sort of match, but not quite. Usually there are a bunch of settings only in the old dialog, and a few new ones only in the new, and some interactions between settings are meaningful, so... you basically just fiddle till it works or breaks.
Any (significant) UI change is bad. Churn is bad. If you want to change the UI because the old one really isn't good enough anymore, you want to do so as infrequently as possible, and as completely as possible: no two-slightly-inconsistent-UIs-at-once. Realize that every change starts with a serious downside and needs a lot of upside to compensate.
So I get that Microsoft wants to update some of the windows settings stuff (and other integrated apps). But the aim must be for a fairly quick transition; no years of overlap. And if for some reason overlap is absolutely inevitable, then sorry, you're going to have to make both systems 100% equivalent. Yes, that means adding features to the old UI, and adding legacy options to the new UI - but at least that way there's a migration path. The alternative is the mess we have now: little point in even having all those settings, because 99% of users are merely going to be hurt by all those options they cannot actually control correctly.
I'd honestly rather have retained some legacy win95 era controls and have only one control for each option, than this mess. If you're going to fix it... fix it already!
Unfortunately, you have to do so so often that discoverability of where things are in the new UI suffers. It's not worth bothering to learn, particularly when the new UI is in flux and changing often.
I'd also like to mention the Microsoft News program as being an annoying delivery mechanism for such content (it seems to have some relationship with the browser based on how the user uses the Internet).
I imagine management in Microsoft is like a fight in the wheelhouse:
Someone trying to make the best OS there is.
Others trying to do ads everywhere like Google.
Some trying to be a reaponsible reliable company, others somehow managing to to crazy price hikes.
Some trying to be friends with open source, others pushing Edge really hard (but, in fairness, still not as ugly as Google was pushing Chrome at its worst.)
Etc etc
But as you said, from sp2 days on it was rock solid and reliable. And since everyone skipped Vista, and a notable amount of people even 7, it was "here to stay". While any previous version of Windows was replaced within 2 to 4 years, many used XP for close to a decade. Some grew up on this OS.
When 7 came along it was a welcome evolution of Windows without breaking too many workflows that relied on muscle memory, for better or for worse.
8 was a clusterduck just like Vista, but for different reasons. Vista fell behind schedule and was unpolished and also, Microsoft overestimated the advancements in hardware power that would be made during its development time. With 8 the problem was that Microsoft went "all in" with conquering the mobile market. The ambitious idea of creating a unified user experience on every platform, while designing every thing as mobile first and then just slamming it onto Windows on the desktop. Obviously they didn't nail this first try, and although Windows phone 8.1 was a really solid user experience we know how this ended. So the primary reason why this modern UI was conceived was eventually gone, but by the time the decision to scrap Windows mobile was made, it was already on the desktop, but at least now you can make changes to the UI without thinking about phones. (just tablets and game consoles...)
With 10 they're still to this day working on creating a consistent modern UX on desktop. I'm still highly skeptical whether that was the right thing to do at all. Considering the current trend everything seems to be eventually web based, and those couple business use cases probably wouldn't care if Windows 10 still looked like XP.
And that all didn't even address the couple major duckups they introduced with updates. Older Windows had its problems, but that's a whole new dimension here. Some ex Microsoft employee claimed on his YouTube channel that Microsoft massively scaled down their own QA. Wouldn't be too surprised if that was the case.
For example, setting your computer’s IP or gateway. Control panel -> Network -> right-click on the “IP interfaces” among a list of 5 abscon names -> Details -> Advanced -> Details -> And you can’t freely type the new IP, you have to use their fields with 4 numbers separated by dots.
Everything was like this. Installing Maven? You need to set a system property, it’s 15 clicks including 6 on a “Details” or “Advanced settings” button. They wanted to hide complexity behind “Advanced” buttons but they hid the wrong things, because each dialog was full of info you’d never use, and what you wanted was inevitably under the “Advanced” button.
Installing new software? Welcome to the installation assistant. We’ll guide you through the 20 clicks, including the mass-unticking of additional software we’ll inevitably select by default, notably the Ask Toolbar when you install Java. If you’re setting up a dev machine you had a dozen programs to set up, each of them requiring a dozen steps of a dozen clicks.
Fair enugh, but depending on what distro you run that seems to be fairly often as well. Like, more than 'a few times' a year.
Then again if you dont run a server for which downtime is a true loss, or something alike, is rebooting that much of a problem (honest question)?
In practice, my companies have done them less frequently. Kernel packages are less often about security patches and more often around a magic combination of kernel+GPU driver+software--I think one of them paid Red Hat for custom cuts.
As for maintaining a personal machine, that distro would probably suggest rebooting more often than RHEL (you'd also have to effectively logout/"restart" for any core package for your Desktop Environment or X). I haven't measured, but I feel like my Synology gets updated every few months.
As for the impact of rebooting. Windows Updates (and macOS updates) are more than just a reboot. It's often an unknown amount of time. Even for a normal reboot, it's really really nice to just leave everything open so you can pick up where you left off. Personal or work it's more than just a browser, but terminals I have open, shell history, vim sessions.
It's like a recent article about fast software https://craigmod.com/essays/fast_software/
This is really a damn shame, because it is far away the best Windows distribution ever. If I didn't have access to it through MSDN, I'd pay a couple hundred bucks for a license.
I still boot an XP partition from time to time. To play old games that simply run with issues in more modern Windows. The issue is XP had the right APIs for game development, and then MS deprecated some great technologies in favour of XBox centered ones.
In new games I still miss the hardware accelerated positional audio XP has.