Re:Shiichan Anonymous BBS (2004)
by noduerme on Sunday August 01, 2021 @10:30PM
(#28032290)
> I'd be interested in pointers to prior art of things which started asinine, and wound up literally deadly.
I've had a running interest for years in deaths caused by bad software design. "Bad design" doesn't necessarily mean bugs (although in rare cases it can). More often, it means building a system without thinking through the real world consequences, or hewing to an unconfirmed, biased view of how it "should" act in the real world. The 737 Max software that tried to deduce an angle of attack with too few redundant measurements, and discarding unlikely results, is an obvious example of programmer hubris. Facebook's news feed is another example of the failure to consider how individual secondary effects from an interface can engender massively catastrophic events at scale.
Cryptocurrency also started asinine, and is becoming increasingly deadly, as it facilitates ransomware attacks on hospitals and infrastructure. Any car manufacturer claiming to be "FSD" right now also meets the definition.
Whenever stories warn us about the dangers of technology, from Daedalus to the Terminator, they're really warning us about hubris. When tinkerers make machines or programmers write code based on their idea of how things should go, or how they're expected to go, based on their personal prior experience or fondest wishes, that's usually when shit goes off the rails.
I've had a running interest for years in deaths caused by bad software design. "Bad design" doesn't necessarily mean bugs (although in rare cases it can). More often, it means building a system without thinking through the real world consequences, or hewing to an unconfirmed, biased view of how it "should" act in the real world. The 737 Max software that tried to deduce an angle of attack with too few redundant measurements, and discarding unlikely results, is an obvious example of programmer hubris. Facebook's news feed is another example of the failure to consider how individual secondary effects from an interface can engender massively catastrophic events at scale.
Cryptocurrency also started asinine, and is becoming increasingly deadly, as it facilitates ransomware attacks on hospitals and infrastructure. Any car manufacturer claiming to be "FSD" right now also meets the definition.
Whenever stories warn us about the dangers of technology, from Daedalus to the Terminator, they're really warning us about hubris. When tinkerers make machines or programmers write code based on their idea of how things should go, or how they're expected to go, based on their personal prior experience or fondest wishes, that's usually when shit goes off the rails.