Somewhere in there, one of the SREs said something that's stuck with me. Roughly, he said that the concept of human error is useless. Every time a human error escalates into a real problem, the deeper problem is a design flaw. At scale, human errors are a certainty, and, while documentation and training is far from useless, no amount of it can change that fact. It's therefore a critical responsibility of systems designers to design their systems to be resilient to mistakes.
https://www.npr.org/2018/08/27/642310810/you-2-0-check-yours...
Oddly enough, the ones who utilized it most were in the military, using the "Swiss Cheese" method of failure analysis. In order for a failure to actually occur, there had to be a complete path all the way through the "cheese", where the problem wasn't caught by any "slice". So a failure at the lowest level was also a failure at each level all the way to the top, and each level was required to figure out why it wasn't caught at their level. Very effective system.
I remember an ex-colleague saying that as an RAF person, he knew he could go on to a totally new airfield and ask for "XYZ Role" officer - and there would be an answer and direction to appropriate person.
Military knows to design also for "deceased personnel" (redundancies etc)...
So many IT outfits have problems that come back to "Oh, that was going to Bob and Bob left" and it infuriates me every time I see it, because we've known for decades how to avoid this problem.
There is a huge problem in the civil service of information being local to teams. Important information sots in inboxes or a shared drive if you’re lucky.
Finding out who to contact in another department on a given topic can take days and usually requires asking your private office to check (because they at least have the other department’s private office contact details). Otherwise people just look for emails in published documents and email out in the hope that at least one of them still works in the right department.
This could all be so easily solved with permanent, shared mailboxes per team. I suspect the reluctance is in part due to how much easier it would make FOIs.
Military needs consistency, predictability, and uniformity. It needs a lot of good management practices to achieve that with people and parts that are not consistent, predictable, or uniform.
Companies tend to focus on chiseling pennies and hope for the best.
Page 2 of this link goes into a useful breakdown
https://sma.nasa.gov/docs/default-source/sma-disciplines-and...
[1] https://cirrusaircraft.com/story/introducing-safe-return-eme...
Of course if you are in an old airplane (private plane, or legally retired but still in use in some poor country) all bets are off.
Is it possible that they could talk a civilian down in an otherwise perfectly functional jet airliner given the pilots both are totally unavailable? Yes, but it'd be really hard and I would not give them great odds for it working.
Plenty of random non-pilots won't even successfully work an airliner's radio, if you end up talking to yourself, or the passenger cabin, that doesn't help you.
In contrast what Safe Return Emergency does is really one button "Oops, the pilot is unconscious/ dead, fix it". It has the same reassuring "Don't touch the controls" you get from Waymo cars. The SRE system is flying the plane, you can best help by staying calm and maybe trying to fix the pilot if that possible. It will figure out where there's a viable landing strip it can reach, broadcast its intentions to nearby humans, fly to its chosen strip, set up, perform the landing, roll out, stop and shut down the plane.
Pushing a button is or course a lot easier. I don't want to diminishing that. However most of us can't afford the type of plane ticket where it is needed, first class on a commercial flight is an order of magnitude cheaper. Thus the odds of both pilots dead is much higher, and so knowing there is still hope may be useful.
[1] https://www.faasafety.gov/SPANS/noticeView.aspx?nid=11667
It would make the error much easier to spot.
OTOH, the people approving may not have the right to see values, just the process part they are involved in.
This and the aerospace mention reminded me of John Glenn: "two million parts, all built by the lowest bidder".
If the person went out of their way to perform the act, for instance they manually turned off all alarms because they wanted to take a nap, then sure, it's (largely) human error. But then you can still ask, why were they even able to turn off critical alarms? Why was there a single person involved in this apparently crucial position? What allowed us to hire someone like Homer Simpson into the nuclear power plant in the first place? And do we really want to rely on eeny-meeny-miny-moe to solve our problems? Maybe training needs to be improved, including evaluation of training results through simulations.
1. First nuclear accident: the theory is one operator was upset that another operator was screwing his wife, and removed the control rods 100% when he was scheduled to work on them, causing a sudden spike in temperature and then a water hammer that killed everyone in the building. (and made it JUMP!!!)
2. Chernobyl, although many people (specially anti-nuclear) blame the accident on the design, it was obvious deliberate error, even if the intentions weren't malicious, the operators ignored alarms, then DISABLED the alarms, and intentionally pushed the reactor past safety limits, even if it didn't had the design flaw it had, it was possible they would still find a way to blow it up if they kept their behaviour.
3. Depressed airline pilot, locked co-pilot out of the cockpit and crashed on the ground on purpose.
There are probably other situations out there... thing is, those are situations no matter how well the system is designed, it will still happen, it is impossible to make a "malicious-proof" system, after all, even a rock, with zero moving parts, can be used in the "wrong way" and kill someone.
On Chernobyl, my take away from the excellent Netflix mini-series was that there was a massive cultural failure, where everyone abdicated responsibility, and there no ability for employees to counter poor decisions from superiors. Culture and organizational design are also important components of complex systems
If you're interested in specific stories I'd recommend "Set Phases on Stun" by Steven Casey. (It shows up in the syllabus of a number of Human-Computer Interaction classes.)
But as a general heuristic: If one operator has trouble dealing with the number of false alarms you can just hire a new operator, but if most of the operators find it problematic then the problem clearly isn't with the operators.
The notable example of alarm fatigue is the designated pillow on board of the Tu-144 (the Russian copy of the Concorde) to stuff into the alarm horn. The wikipedia page is itself a “bestof” bad engineering/operating practices, which are worth a good laugh now that it is not flying anymore. See chapter “Reasons for cancellation” - https://en.wikipedia.org/wiki/Tupolev_Tu-144
Most telephone hardware has some form of self-checking and fault detection. Alarms are classified into "minor" and "major", sometimes also "power" or "critical". Each piece of equipment has its own alarm indicator lamps, and contacts for external alarms as well. Typically the external contacts are wired to both an "aisle light" which aggregates all the equipment in that lineup, and a floor-wide or section-wide "alarm sounder". When the sounder goes off, you look down the aisle to see which lineup has the trouble, then walk down the lineup to find the individual unit and address the problem.
But first, you push the ACO button. The alarm cutoff button silences the sounder, which won't come back on unless something new happens. ACO also tells the other workers that someone is on the case.
Crucially, it preserves attention. If the sounder was going the whole time someone was working on a repair, nothing would ever get done, because the workers would be down at the corner bar trying to get some sanity back.
The designer responsible for an alarm wants to avoid being blamed for the operator missing that specific alarm so they're incentivized to make their alarm as prominent as they can get away with. Of course once most designers are doing that then they have to keep doing it or risk get drowned out.
One way to fix that is by having someone who is responsible for the entire experience who can dictate shared prioritization, requirements, alerting frameworks, rate limiters, etc. In cases where the operator uses multiple products from different companies (e.g. in a hospital) that's especially hard which is why there's a cacophony of noise even though everyone involved acknowledges that it's a problem.
Somehow this works just fine in telecom. I was handling equipment from DSC, Alcatel, Fujitsu, Marconi, Rockwell, Cerent, Nortel, Pirelli, CarrierAccess, Tellabs, Cisco, ADC, Lucent, and more. There are Telcordia and NEBS standards for alarm severity and wiring, and everything just works when you hook it up. Does the medical industry not have standards?
Ref:
https://www.ecri.org/alarm-safety-handbook
https://www.jointcommission.org/resources/patient-safety-top...
Everyone does it. I’ve caught myself doing it. So I move to the previous domino and work on that.
Red lights mean that the sensors have detected a state that may damage the car or put you in danger. You should stop immediately. Examples: low oil pressure, overheat, emergency brake still on, low brake fluid. You can safely tell a new driver that if they are driving and they see a red light on the dash it means stop right now.
Yellow means that something is operating outside of nominal ranges and you should try and fix it soon. Examples: low tire pressure, low washer fluid, check engine light.
Link: https://www.youtube.com/watch?v=1xQeXOz0Ncs
One thing it discusses is how the alerts inside the control hub were so noisy, and were logged via a reasonably slow printer, that the alerts which suggested the reactor was in serious trouble only printed hours after you could do anything.
And the alert light that said "shits fucked" was placed right next to the light about an elevator being out of order.
Human error is rarely intentional, and even if it is, a system that fails in response has missed the mark. The talk explains the amazing finds made by the government committee hired to investigate the incident, once they left the idea of 'human error' at the door.
I do suggest you watch the video, but as a teaser: it turned out that at the time, most US nuclear engineers had been sourced from nuclear subs. Subs have totally different rules on how to manage the reactor, and train their staff inappropriately for a land-based reactor. Pretty significant cultural find...
That's _one_ theory and it's not even the most likely one, it's just the only one they like to trot out whenever a History Channel producer needs to fill time between ads.
The control rods in SL-1 were prone to getting stuck in their channels for a variety of reasons. They were apparently not light either, as part of the investigation for the accident, they built a mockup control rod assembly that was weighted at 84 pounds. For military-age adult male, this is right in the range being able to move, without a lot of control over how _much_ it is moved.
It is, however, clearly a design failure that the removal of a single rod allowed the reactor to go critical, and/or that the rod was able to be removed by a human to the extent it was.
"Doors can have two states: open or closed. They should not require instruction."
Also there is that fun state of “open but will lock behind you because the doorknob lock does not prevent you from closing the door, hope you have your keys on you or someone inside” though that’s maybe more a special case in the state transition table than an actual state?
How could we tolerate such an awful design? Who is responsible for this catastrophe?
I don’t think a door people regularly walk into out of confusion is a good one.
Sadly.
* ie: Defend (with passion!) the flawed tools, and resist attempts to fix or change them by something better
For me, the best illustration is the Tenerife airport disaster, the deadliest accident in the history of aviation. Two Boeing 747 crashing into each other on the runway during takeoff.
Circumstances included fog, problems in the control tower, nonstandard phraseology, radio interference and overworked personnel.
For example, let's say a pilot belly-lands because they failed to lower their landing gear, possibly because they were distracted by some other equipment failure, or possibly because sometimes qualified and experienced pilots do this. This error could have been avoided with, say, an automated spoken "gear up" warning instead of a nonverbal warning horn (I have no idea if some aircraft already have this). But it could also have been avoided by longer and more extensive training, right?
Sure, it maybe could, but why risk it? Humans are also humans, and even pilots with endless amount of flight time does mistakes sometimes, because we're all humans. The idea is to work with the idea that sometimes we fail, even if we are well trained, so when failures happen, it's easy to recover.
> Where do you draw the line between blaming the UX and blaming the training process, or the inevitable cases where an irreversible decision made under uncertainty turned out to be the wrong one?
Basically, it's always the UX. If there is a case where the pilot made the wrong decision, something needs to change so if a pilot with similar experience, wouldn't have made that decision with a different UX. Basically force the pilots to do the right decision, always.
I'm not 100% on the history of all of this, so if someone is more familiar with it, please correct me. But I think it started with "Crew resource management" to ensure proper training of the crew. It has gone through many iterations where less and less blame is being made on the crew and more on the equipment. I think the point of the current iteration basically boils down to "Human-error will happen, let's plan for it and if human-error lead to a plane going down, let's fix it so the same error would be prevented".
That's not realistic. If you think it is realistic, please take a look at the investigation report for PK8303. Some pilots will disregard alerts, override safeties and ignore procedures: there's only so much a system can do to promote good decision-making.
https://www.caapakistan.com.pk/Upload/SIBReports/AAIB-431.pd...
If it were possible to do this, we wouldn't have pilots at all.
Generally in this sort of risk analysis there is a hierarchy of mitigation, and the least of them is training. e.g. when you find a potentially hazardous situation, the best thing you can do is design the system so that it can't happen. Failing that, you design in an procedural or workflow aspect that naturally avoids it. Failing that, you put in a system to force the use to confirm or review the action. Failing that, you train around it.
A common sign of a flawed design process is when things get pushed into the "train around it" bucket late in the process, not because they were unavoidable, but because they were looked at too late in the design to effectively make changes.
I view the procedural or workflow aspect as a subset of training. As an example, I always lower the gear and leave my hand on the gear selector until I see three green (gear safe) lights prior to selecting any flaps.
The operating handbook and factory checklists do not require this, but my planning and training has elected this optional workflow/procedure. Why? Because it's incredibly hard to get my airplane to slow down and come down for landing while completely clean (no flaps, no gear). If I use the factory accepted procedure to select approach flaps, I can get it slow enough to land. I've literally never seen a "pilot forgot the gear" airplane sitting on the runway with no flaps deployed.
Example: I have a mechanical system moving fast with enough mass to hurt the operator, so I put the control on the other side of the machine (i.e. machine body between person and the dangerous part), and require the operator to hold a switch "on" for the entire time the machine is moving.
This doesn't make it impossible for the machine to hurt the operator, of course, but it makes it a lot less likely - and it doesn't rely on training at all.
While you can also rely on training aspects like you describe (a good idea!) it's not the same thing.
I don't know aircraft, so I don't know if there is something equivalent but I wouldn't be surprised. Anything you simply have to do with two hands on disparate controls?
The only thing that I can think of that directly work this way are some of the gust locks (to prevent control surfaces from banging about in the wind while parked) are required to be designed so the airplane can’t be started and taxied out with them in place. (That’s arguably a poka-yoke as much as a procedure, but probably still fits.) Mine simultaneously locks the yoke and has a plastic flag that blocks the starter switch when installed.
If the airplane offered a mechanism to inhibit flap operation unless the gear was down and locked (a poka-yoke enforcement of my procedure), I’d decline it/argue against it. In the event of a gear failure or a forced water landing, I want to have the flaps without the gear being safe and, especially for water, I don’t want to have to look up the abnormal/emergency flap extension without gear in a QRH while doing everything else needed while descending for a forced landing.
In my aircraft, it's triggered by ((no gear safe indication) and ((full flaps) or (very low throttle setting))). The presentation is a loud, alternating warning tone and a red "Gear Unsafe" light directly in my field of view on the glareshield. These are not silenceable by button as is common on training aircraft.
Even though a handful are inadvertently geared up every year, I'm not convinced that better warning systems would significantly reduce the rate. Better training would. If I ever do the deed, I'll be the first to admit that I was the primary cause and that I failed not only to follow the checklist, but my own SOPs and flew the aircraft onto the runway with a large red light right in front of me and a blaring horn. Making it a voice seems not likely to help.
If I gear it up after an electrical failure [which some of the inadvertent ones are], that's probably slightly more understandable (and a scenario in which an additional, electrically powered, warning system would presumably also be INOP).
If only the people who insist on using unsafe languages understood this. Their position is that you should get better at not making mistakes.
Anyone who thinks they can overcome 100% of memory-unsafety bugs through looking at the code real hard is fooling themselves. But I'm not convinced there's a lot of people who actually think that (maybe there's some people who just don't care or never considered it)
It's written by someone that does airliner crash investigations. His central point is that "human error" as a term functions to redirect blame away from the people who establish systems and procedures. It blames the last domino vs the people who stacked them.
It's a quick breezy read, and you'll get the main points within the first 30 min or so of reading. I've found it useful for getting these ideas across to people though, especially more generic business types where "no blame post mortem" strikes them as some care bear nonsense rather than being an absolutely essential tool to reduce future incidents.
And then proceeds to use reductio ad absurdum to demonstrate that if you can be perfect for 5 seconds, you can string those together to be perfect for 5 minutes ... 5 hours ... 5 days .. your whole life!
And when someone in the crowd pushed back on this idea, the consultant said something like, "If the airlines took that same approach to flying, we'd have 1,000 accidents a year!"
And then maybe the reader looked it up later and it turned out there were about 1,000 flying accidents a year.
Had the social structure been politically different on that day they would have acted on the available engineering information by refusing to launch. It was the political environment that goaded them into taking more risk than they had previously decided was acceptable.
Obviously they may not know these conditions ahead of time. But preventing user error for known problems seems obvious.
The problem is Citi itself. It's always an absurd amount of indian subcontractors who aren't paid to care but just to look good on a cost cutting plan (so they do their job), a disorganized / panicked organization who seem to never know what's happening when and follow mega-processes rather than solve problems.
My bank has those, but we can take shortcut if we must - not Citi, can make you wait 2 months for a csv file via sftp, and then pretend to be ready when you finally discover it's a frigging indian contractor writing the csv rather than an automated process... we were like "guys seriously why are you even lying about it" when it all came crashing down with crazy errors never seen in the UAT. Turns out the automation guys were late and they made an effing human do the prod version meanwhile ...
In another company, working with Citi on a completely different sector/group/vertical, I spent nights with their tech guys in India figuring out their own system and why they couldn't work... and they had less clue than me, it's absurd.
They seem to work in very split team where americans will do the high level design and then outsource to dev + support + deployment teams, ofc isolated from each other, in India. So nobody US cares about any actual implementation detail of anything, the 3 indian teams don't know each other, and nobody feels responsible of anything. It's absurd I've had multiple separate experience of the same thing, with me and my colleagues doing the connection layer between various Citi stakeholder who had no idea about each other.
There was even a time where a Citi country team in Hong Kong begged us to introduce them to the Citi country team in Singapore, and show them their product and processes...
We've got four levels: info, warning, severe warning and error. Warnings are highlighted, severe warnings must be explicitly accepted (and we log this) and errors prevent submission.
When clicking on an item in the list, the cursor moves to the relevant control, so they don't have to hunt around.
We add the checks we can think of, and keep on adding checks based on user mistakes and feedback.
I'm not familiar with the financial world, so I don't know how common it would be to do what they wanted to do, but at the very least it sounds like the principal account number being filled out but not the two others is something we'd made an info item from, to highlight what would happen. If what they ended up doing is unusual, it would be a warning and likely severe warning due to the amount.
However, a dev that does understand action B would be able to anticipate various forms of input from button A and how that could cause undesired results from action B.
This isn't a knock on the dev that is less knowledgable about the full thing. It is a knock on the people upstream that should be more aware of issues. Also, the people most knowledgeable about action B tend to have little to no understanding of programming. While they might understand that providing bogus data to action B can have bad results, they are not aware enough to tell the devs that sometimes valid looking data is just not sane.
I assume this system is designed to handle manually adjusted payments for syndicated lending. So you've got one borrower with potentially multiple credit facilities, with each credit facility will be funded by multiple lenders, each with potentially multiple accounts. The events you are dealing with are fees, underpayment/overpayment of interest or principal, late or early payments of principal, transfers of balances from one lender to another, funding transactions and plenty I haven't thought of or considered but are represented by check boxes on that UI.
It's complicated. The reason the system exists is because it's complicated, and prone to human error, and the before and after might generate dozens of events in dozens of accounts.
They all thought they were making the right transaction. There was no warning that they were not.
It would be like if I hit the reply button here on Hacker News and instead of the comment I typed out it my reply contained my bank account information. And I was not told something I do not want to happen would happen by doing so.
Thanks for the explanation. It saddens me how brutal people here are in response to a misunderstanding, judging by the downvotes, but oh well.
> Citibank's procedures require that three people sign off on a transaction of this size. In this case, that was Raj, a colleague of his in India, and a senior Citibank official in Delaware named Vincent Fratta. All three believed that setting the "principal" field to an internal wash account number would prevent payment of the principal. As he approved the transaction, Fratta wrote: "looks good, please proceed. Principal is going to wash."