We were a US company working with a Japanese software distributor to do Japanese versions of our products. Occasionally on some Japanese non-IBM compatible PCs [1] we were seeing a lockup during installation.
It was the kind of lockup where CTRL-ALT-DEL does nothing, the CAPS LOCK light no longer toggles, and if you have a GUI that mouse pointer no longer moves. There's usually pretty much nothing to do at that point except hit the reset button or toggle power.
It was quite rare, giving us not much to work with. Our Japanese partners decided it was rare enough to go ahead and ship handling the (hopefully) handful of people that hit it via tech support.
So we shipped. And they got something like 100 support calls--but the callers were not upset. In fact they were happy with the product except that they wanted to suggest that the installer should be made faster or should run in the background so they could use the computer while the install takes place. The reports said that the install took something like 20-30 hours.
I can't even conceive of a US consumer letting a computer that appears totally locked up while installing a small utility program sit for up to 30 hours.
(BTW, knowing that it was just very slow rather than frozen was enough of a clue to let us figure it out. We needed to scan for hard drives and CD-ROM drives. The way we were doing that was hitting some sort of edge case on a particular model of controller's firmware that made some requests take a long time. We were able to change the drive scan to avoid that controller's edge case).
Seriously now...Hearing the pain here frustrates me because i am actually in favor of paying my taxes because i know when handled properly then roads, bridges, infrastructure, other things, and yes, the right tools for our military are done then everyone benefits...but when things are not managed right, it chafes me so much because the pain ripples all the way through from civil servants unable to do their job, to military folks not having tools to help defend us, to civilians being put at risk while going over unsafe bridges, to commerce not being efficient because roads suck, etc.
Just from reading your comment, sounds like the biggest problem is the vendor overlap here. MS Endpoint Manager, MacAfee and Tanium all seem to be doing the same thing.
This seems like scope creep on an epic scale. It's almost like someone came to USAF IT every 3 years and said "we signed a contract with X vendor, add it to the laptops".
Given what I've seen of corp/gov IT, it wouldn't surprise me in the least if the "security software" (aka virus scanners, intrusion detection, etc) then proceeds to hammer the disk/CPU sufficiently that the machine cannot make forward progress. My wife's work machine (a fairly nice, if slim dell) sits there with one of its 4 cores at 100% running a McAfee process hammering the disk doing some kind of scan pretty much 100% of the time. The result is that it pushes that core to max turbo, overheats the system and then throttles the whole thing to ~800Mhz. It then takes a minute or two to open word/etc. And that is with a fairly fast SSD, add in the spinning rust, and i'm sure it would take much longer.
And of course its locked down sufficiently that its not possible to even try and fix it.
Scanning through the replies, I'll also add that if you are loading your entire profile from the network, this often becomes much more problematic than the IT people may have initially calculated, because even if you can transfer a gigabyte in 10 seconds, you can't transfer a gigabyte in 10 seconds to thousands of users simultaneously, and logins are highly temporally correlated. It doesn't take much at all to have a cascading network failure as a result, even under perfect conditions, as that 10 second window becomes a 20 second window, which catches ten more people and it turns into a 40 second window, and it explodes quite fast from there as suddenly everyone everywhere is waiting for minutes or hours to load from the same overloaded profile server(s).
From what I've witnessed of this, what you'll get is a number of these answers operating at once.
I found out this is a thing when Chrome and Firefox started to put their files in the user's home directory by default. If that has to traverse a network - and who knows where all that data is coming from or how fast the internet connection is - that can take a long time.
The idea that you can log into any PC and your local desktop and documents all appear seems really nice, but the implementation weeds are nasty: I tell people to store things they want to access on multiple computers on a network share, and we replicate basically nothing between client PCs.
Outlook also likes to download a large chunk of a user's mail archive locally each time you open it on a PC, but you can configure it a bit, and Outlook generally "works fine" while downloading. I suspect it gets hairier at DOD scale though.
At my company people would remotely log in on Sunday evenings to avoid the hou long wait while a few hundred gigs get copied across the network. Why not chnage the location of the local repo, you ask? Of course our machines are locked down and we have no access to the local drive...
There's lots of reasons it can happen though (especially on Windows, where the mood for 20 years was "cram it all into apply on login GPOs and scripts"). My guess in this case since the machine costs $109 (used, I assume), is that it is running spinning rust and 4GB of RAM, which on Windows 10 isn't a lot of fun. Couple that with all the security crap the DoD has to put on there (which probably spends half its time trying to grab resources from the other security software) and I could definitely see an hour.
Corporate spyware is a huge impact to performance, particularly if you have an HDD. I can’t count how many times I’ve seen a computer start to lock up because some security software needs to absolutely abuse the disk.
Outages/workstop are pretty binary. Otoh its hard to see how much time people spend just waiting for stuff to complete. At my last job, pulling down a copy of the test db for local use was taking 30-40 minutes. But because everything else was slow, it was only after I noticed it being a lot shorter for coworkers that I even bothered investigating what was causing it (version of mysql was old, made it a 5-10 minute process after fixing a config).
I’ve got a hint that a lot of it is doing stuff that the OS might even consider malicious.
Then they also inject themselves into processes and 'hook' into operating system routines to check things the feeds to not provide: which files are being accessed, how often, what memory is allocated, what type of memory is allocated, which threads are running, where they are running and what they are doing, if there's mismatches between what the operating system told are the modules loaded, and what it can find in memory.
Most of this software can be configured so that the resource usage is relatively tame, but then on the next pentest the security people will notice all sorts of ways the products did not catch them (this is the usual case). And then things are tuned to max in short order :) And then you have security software running multiple rules and scans on each file any process opens. And processes open a lot of files, all the time.
I didn't know what this meant, so I looked it up on DuckDuckGo.
The first relevant result was from... a DoD website.
You do not want to apply all of their reccomendations to any computer, but I will say that fully STIG'd computer will actually not take that long to boot up, aside from the first application of the policies. Login performance usually comes down to drive maps, printer maps, and startup programs.
>Currently the service uses both McAfee and Tanium software packages to scan and protect service-issued endpoints like laptops. But the computing power required by multiple programs often interferes with the user’s work, and damages the user experience, or UX.