He also noted that a lot of American firms worked with Indian firms to resolve it. Indian engineers were well suited to the problem, according to him, because they had been exposed to a lot of the legacy systems involved during their studies.
He said an interesting consequence of this was that many American companies concluded they could use lower wage Indian engineers on other software projects so it helped initiate a wave of offshoring in the early 2000s.
I couldn't find the story I heard in the NPR archives, but they do have a number of stories reported right around that time if you want to see how it was being discussed and treated by NPR at the time:
https://www.npr.org/search?query=y2k&page=1
I did find this NPR interview with an Indian official from 2009 that notes Y2K's impact in US-Indian economic relations:
But I think the real turning point in many ways in the U.S. relations was Y2K, the 2000 - year 2000, Indian software engineers and computer engineers suddenly found themselves in the U.S. helping the U.S. in adjusting itself to the Y2K problem. And that opened a new chapter in our relations with a huge increase in services, trade and software and computer industry.
https://www.npr.org/templates/story/story.php?storyId=120738...
This created an exploitable opportunity for the few who knew how to write automated tests.
During the Y2K panic Sun Microsystems (IIRC) announced that they would pay a bounty of ~$1,000 per Y2K bug that anyone found in their software (due to the lack of automated tests, they didn't even know how to find their Y2K bugs)
James Whittaker (a college professor at the time) worked with his students to create a program that would parse Sun's binaries and discover many types of Y2K bugs. They wrote the code, hit run, and waited.
And waited. And waited.
One week later the code printed out it's findings: It found tens of thousands of bugs.
James Whittaker went to Sun Microsystems with his lawyer. They saw the results and then brought in their own lawyers. Eventually there was some settlement.
One of James' students bought a car with his share.
It was money for jam.
For example, at a former employer of mine, a big justification for getting rid of the IBM-compatible mainframe and a lot of legacy systems which ran on it (various in-house apps written in COBOL and FORTRAN) was Y2K.
In reality, they probably could have updated the mainframe systems to be Y2K-compliant. But, they didn't want to do that. They wanted to dump it all and replace it with an off-the-shelf solution running on Unix and/or Windows. And, for reasons which have absolutely nothing to do with Y2K itself (the expense and limitations of the mainframe platform), it probably was the right call. But, Y2K helped moved it from "wouldn't-it-be-nice-if-we" column into the "must-be-done" column.
> In reality, they probably could have updated the mainframe systems to be Y2K-compliant.
Legacy systems weren't always mainframe systems and, in any case a “legacy system” is precisely one you are no longer confident you can safely update for changing business requirements.
A change in a pervasive assumption touching many parts of a system (like “all dates are always in the 20th century”) is precisely the kind of thing that is high risk with a legacy systems.
Story time: years ago (2010-ish?), we were doing some light web CMS work for a client. Nothing too complex, except we had to regenerate one section off a data feed that we received in a very peculiar custom format.
Great, everything worked, we finished on time and on budget. And then: "oh, and could you also check this other site? We need some minor tweaks on what your predecessors made". That other site was also consuming that data feed, so we took a peek. It ran in PHP3, walled off from everything, processed its own intermediate languageS, and the output was a massive 200+ page PDF (which was then manually shipped to offset print in large quantities). For Reasons, this had to run daily and had to work on first try. There was no documentation, no comments, and no development environment: apparently this was made directly on prod and carefully left untouched.
Needless to say, the code was massively interdependent, so that the minor tweaks were actually somewhat harder. We did manage to set up a development VM though, and document it a bit - but last I checked, the Monstrous Book Of Monsters Generator still seems to be chugging along in its containment.
And don't get me started on Oracle refusing to provide updated y2k compliant backports!
Sounds like some of the benefits from the Coronavirus crisis, like the increase in working from home.
I wonder what the practical mixture of NAND flash vs EEPROMs was. I understand NAND wasn't especially stable back then.
Once upon a time we got somewhere in the neighborhood of 1200 dell workstations and not a single one failed the bios upgrade. 10/10 would do again
Yes, updating JDEdwards AS/400 systems and many a PC was a big project, but i dont recall doing it at that time as being super difficult... frack ist so long ago i cant really recall many of the details other than reporting daily on the number of systems updated.
Also, fuck you google - this was when you were nascent a i converted the entire company to your “minimalist” front page vs yahoo, and a few years later you wasted 3 months of my time interviewing to tell me to expect an offer letter tomorrow and called to then reacind the offer (as i didnt have a degree) and then continuing to contact me for FIVE FUCKING YEARS for the same job that you wouldnt hire me for.
/off-my-chest
None of those people contacting you, or phone screening, were Google employees. They were, or worked for, contractors. It didn't matter to them or to Google whether anybody they contacted was hired.
The pressure was immense. People don't realise what a huge success story the Y2K situation really was.
Whole businesses would be ground to a halt to stem the mess of bad data and if not prepared they would not have quick fix.
In many cases in the age of apps and web apps we can roll out fixes fairly quickly, but in 2000 that was very much not the case for most situations.
So yes indeed stuff was broken, but it got fixed before the big date.
More importantly, lift systems are not sensitive to long duration dates. They do not need to be.
This story reflects one of the myths that emerged from the media. Around 1997 they started discovering the topic of embedded systems and that many devices, including lifts, contained "computers." Not understanding the restricted and specialized nature of embedded systems, they then claimed all these systems were vulnerable to Y2K, and that the whole world was about to crash.
Most embedded systems, in lifts and other devices, had been designed in the 1990s. If they used dates, they did not use mistakes from the 1960s.
The system was probably designed in the 1970s.
This is especially important if you think about what things were like in the late-90s. As a teenage geek in Northern Ireland I read about the the first tech boom, but locally there was very little exploitation of tech and you were an anomaly if you had dial-up internet at home. There was very limited expertise available to fix your computer systems in the best of time.
Big companies had complex systems, but they could also afford contracts with vendors in the UK mainland, or Eire, to fix systems. It was much more limited for other companies. While people were generally not relying on personal computers too much, small businesses had just reached that tipping point where Y2K would have been painful. As a result, they took action.
I only got to visit the US a few times in the early-nineties and it seemed so futuristic (I got a 286 PC in '93 to use alongside my Amiga). I imagined the Y2K problem as being much more painful in the US, and I wasn't accounting for the distribution of expertise.
One great experience was summer training courses that the government had organised with a IT training company in Belfast. These were free and it was a like a rag tag army of teenagers and older geeks from both sides of what was quite a divided community. The systems we were dealing with were fairly simple. It was mostly patching Windows, Sage Accounting, upgrading productivity apps (there was more than Microsoft Office in use) etc., but the trainer was normally teaching more advanced stuff like networking and motherboard repair so we spent a lot of time on that.
I agree with everything you and the parent said, but there was also hype and hysteria on top of it all - especially among the communities of preppers, back to the landers, and eco-extremist types. I met entire communities of technophobes who declared I was just ignorant or 'a sheep' and refused to believe me when I tried to explain that the industry was working on the problem and had a solid shot at preventing major catastrophe.
Frightening predictions of power plants going offline for days, erratic traffic lights causing mayhem, food and water supplies being interrupted, even reactors melting down were not uncommon.
As always, there is money to be made by exploiting people's fear. This hysteria was considered an irrelevant side show by most educated people, the way I look at the chem trail people or the way I used to view the anti-vaxxers [1], but the unjustified hysteria was real.
[1] Before I learned their numbers were growing and could threaten herd immunity in some areas.
https://www.youtube.com/watch?v=x0bBYK-7ZiU
Oh, and Dick Clark melts.
And a few people did take it really seriously and spend the Millennium in some remote place where they would stand a chance of surviving the collapse of Western Civilisation.
this left me at a crossroads. i thought about writing malware that would randomly raise peoples semester grades by 4 points (e.g. C- would became a B+). i thought about mass changing grades. i thought about altering a target group of kids i didn't like. all but the last of the scenarios ended with me getting fingered because I was the smart computer kid. if i didn't touch my grades i would have plausible deniability. i wrote the malware. then i watched office space and decided to think about my actions (and promptly forgot as a horny teenager does). soooo glad i didn't release it because years later i went back into the code and found a leftover debug that would have targeted the only 2 kids in the school who had this letter in their last name.
tl;dr: office space is real
The sceptic inside me is curious as to how much they actually accomplished in their effort to fix it. I mean, yes, they spent billions to get millions of lines of code read, but how many fixes have been made and what would be the cost (when compared to those billions of dollars spent) if they weren't fixed at all.
With an auditorium full of others who had barely any programming experience, he was given a crash course in Cobol for a month, and then he had to go through thousands of lines of code of bank software that ran on some mainframe and identify all cases where a date was hard-coded as 2 characters.
Extremely tedious work, but there were tons of fixes to be made.
The problem was real, and the billions that were spent on fixing things were necessary.
EDIT (for clarity): that’s what I would say, and I’m pretty sure many people smarter than I had said it - so I’m just curious what objections made business push forward with the “relatively unskilled” human labour intensive practices instead, because that’s the history we have now.
Also, much of the remediation could not be automated anyway (for example you may not be able to assume the century is '19' for a given date field which can be in the future. You may need to do a comparison to a cutoff date, or you may only have to enlarge the field knowing that the value is supplied from elsewhere which will itself get fixed), so it fell back to people looking at the code, understanding it just enough, making the required changes, and planning/executing the testing.
Remember that big old systems were the low hanging fruit in 1970. The business logic was COBOL and assembler, without things like functions. The system may have hundreds of duplicate routines for things like subtracting dates.
If you asking "would it be better to have done nothing?" then I would have to say No, of course not. What are you thinking?
I worked on fixing Y2K bugs at the time. It was very methodically and planned, and tested on-site at a space agency. If we hadn't fixed those Y2K issues at the time then weather forecasts would have suffered greatly. The cost of that happening? Hard to calculate I think.
Why are you sceptical about this?
"look at y2k: we spent so much money, and nothing happened!" (so far, so good) "...therefore, if we hadn't, nothing would have happened anyway!!" (uh, based on...?) "...and it was all a massive scam, just like any massive present issue!!!" (ah, there's the underlying reason)
The fact that people worked on date issues for a decade does not mean the public scare campaign in the late 90s was justified.
You could equally claim airplanes would crash if they weren't refuelled, or people would die if garbage wasn't collected. In those cases, the media is educated enough to know the claims are pointless.
In the case of y2k, they had little understanding of how enterprise IT worked, and lots of businesses who were happy to encourage that ignorance.
The Y2K bug did not become a crisis only because people literally spent tens of billions of dollars in effort to fix it. And in the end, everything kept working, so a lot of people thought it wasn't a crisis at all. Complete nonsense.
Yes, it's true that all software occasionally has bugs. But when all the software fails at the same time, a lot of the backup systems simultaneously fail, and you lose the infrastructure to fix things.