1. Thanks for the awesome TIL
2. Rumors fly about old versions of Windows, OS/2, etc still being actively used. I like to pin down and file away usage/year correlations, where possible. What sort of timeframe (roughly) was Win98 in active use here?
3. Regarding [1], I have an ancient [runs downstars to check] Compaq Prosignia 300 server here and I discovered in the (DR-DOS based) BIOS at one point that the serial port's electrical behavior can be customized between being edge triggered or level triggered. (Mildly interesting machine. Insists it has 83MB of RAM. Has the FOOF bug. Its SCSI disks make nice noises when they spin up.) Maybe this is related.
2. The last I had heard about that machine, specifically, was around 2011 and I'm fairly certain it was there when they eliminated the NOC in Detroit which was around 2013, if memory serves. The thing is, I would be surprised if it was actually gone.
3. Nice - I was well known was the guy who could fix anything over at Level 3 around the IT side of the house. Around 2014, I was asked to take a look at an ancient desktop PC that had been sitting in the server room at another building. It had failed weeks prior, had a modem plugged in and was discovered to have been used for a billing purpose that was apparently costing well into the 6 figures. The VP asked me if I would spare a moment to see if I could figure it out, despite it having nothing to do with my normal duties. It was dusty as heck and wouldn't boot -- nobody on the server team could get it functioning. I noticed a home-made label stuck to the top and recognized it as drive geometry. The little battery had failed, probably 15 years prior, and someone put a label on it figuring that people would understand exactly what it was. I was the only one who remembered what that was. When I made it boot, people stared at me like I was a magician. :)
Many years ago I happened to find an ancient-looking machine buried in a spare room at a church. I think the room was occasionally used as an ad-hoc creche area.
After finally locating an IEC cable for it and finally getting it to boot, I found that it was of the opinion it didn't have any HDDs attached.
So, I went into the BIOS, and - yes! Just old enough to require manual CHS configuration, but juuust new enough to have a manual autodetect routine!
Turned out it was a cute 25MHz-or-so (IIRC) 486 with something like a 200MB HDD. Had some demos on it that I've long forgotten the names of. Was fun to find that machine.
Despite being so trivial that no conveniently-placed homemade rescue labels were needed, the people there also wondered how I'd figured out what was wrong with it as well.
(Honestly, I really want to work somewhere everybody's still using ancient equipment. Partly because it's what I've been exposed to for most of my life and I really like it, and partly because I'm still yet to have the chance to acclimatize to newer stuff and almost all of the small bit of knowledge I've accumulated covers older tech.)
Regarding what you helped me figure out, I've been wondering for years why that BIOS decided it didn't have a HDD. Initially I thought the HDD was on the way out and decided not to show up one day, and the BIOS happily deregistered it [after someone saw an indecipherable POST error and hit F1 or whatever]. Now I wonder if maybe the rechargeable battery went flat after the machine was left off for ages, then someone turned it on, hit F1 to accept the "bad CRC" error (I never saw one) and didn't know to do an autodetect for the HDD. It's possible. I know some BIOSes remember "time wasn't re-set after fail"; I never saw a time POST error, and don't remember if I checked the clock to see if it was wrong.
Things like that make John Titor's IBM 5100 story slightly more believable. Old computers often have strange features which are omitted from their successors due to being rarely used (for instance, the original IBM PC could use current loop on its serial port connector). One could imagine someone with access to a time machine taking a short trip to the past just to grab one of these old computers as a replacement for a failed one.
Nokia also started as a rubber company:
https://en.wikipedia.org/wiki/History_of_Nokia
They got into telecommunications by buying a cable works. I don't know if that was just random empire-building, or if there are material connections between cables and rubber. I suppose you need a lot of rubber to insulate the cables.
Although agree that Cisco is usually pretty solid.
The article and at least one other commenter mentioned this being a UI problem, and all I can say is "bingo". They didn't identify the vendor, but the article called out Cisco. I am a little skeptical of that, personally[0]. I lean toward it being something else, mainly because of the statement that followed "no one at Level 3 was aware of the consequences of leaving that field empty". We had a lot of very knowledgeable Cisco folks there. There are a few folks that I knew personally that were probably among the top folks on administering that equipment outside of Cisco. In addition, if a problem like this arose, they're accessible and helpful.
It was almost certainly one of the many, many, outdated software applications that make up the vast array of management interfaces into the equipment. I worked inside one of the phone switches (technically, one of the test switches). About a year prior to my leaving, the room next to my desk was filled floor to ceiling with cards that had been there on the day I started. I was told these cards ran along the lines of $30,000 a piece. If you wanted to upgrade the software, you had to replace the card. So we had machines in our operations center that were on isolated networks running Windows 98 in order to run the executable required to configure the switches[1]. We had devices made by Pirelli[3] that had similarly awful software. I knew of about 5 different devices with 5 different problem management tools but there were several more. And there were the handful of devices that were well known as ones "you don't even breathe on" (and I've never heard those words said about the Cisco products ... not that people spoke terribly highly of them, but never quite that negatively).
Telecom went through a really bad time in the early 2000s. At Global Crossing, every 6 months about 10% of the staff was laid off ... this happened for 10 years. Hardware didn't get upgraded, and therefore software didn't get upgraded. Sofware support contracts and maintenance were allowed to lapse. The quality of the software -- since its audience was a small handful of companies, many of which in the same financial state as Global Crossing -- was awful. After that many layoffs, the one or two guys who knew the myriad of corner cases involved in operating some of the management interfaces were on to other jobs at other companies, or retired. That this sort of thing didn't happen somewhat regularly is still surprising to me to this day.
At the time that I left, Level 3 was doing better from a financial standpoint and money was being invested in modernising a lot of these problems, but it's a huge network with many switch sites[4] in any major city that Level 3 terminated wires. Each of those sites had an array of equipment, some of it common, some of it from companies that went under shortly after the dot-com bust.
[0] It could easily have been a Cisco product, but there are so many -- far worse -- products out there that nobody outside of telecom has ever dealt with. I lean more toward that.
[1] Before someone says "Why didn't you virtualize them?". That was done for some machines with both running inside the isolated network, but it became more of a hassle than it was worth since these machines had to occasionally connect to the devices via serial port. The very buggy software included a very temperamental driver that only worked with a few models of older PCs and IBM (not Lenovo) Thinkpad laptop serial ports ... and only then if one of the wires was cut on the cable.
[2] Earlier than the style recently featured in a post about enthusiasts for the brand upgrading the motherboards.
[3] https://www.pirelli.com/global/en-ww/homepage ... Yes, that Pirelli. There's crossover between the rubber tires and fiber optics, apparently ... that was news to me. And they had some of the worst software I've ever encountered.
[4] We had, I think, 4 in Detroit. Some are not in terribly convenient locations, either. One in the area required passing an expensive OSHA certification, wearing steel toed boots and a hard hat if you wished to access it due to its proximity to a rail yard. I'd been there once and couldn't imagine the need for the restrictions.