Re:Ask HN: Share your dev horror stories
by Jtsummers on Wednesday October 25, 2017 @12:56PM
(#15551778)
First job, I'm going to be vague as we did fix it but it was still rather horrifying to be a part of.
I was in V&V at the time, not yet a developer (professionally). We had a maintenance task, an old system that was built some 15+ years before any of us [0] had started working for this company. Fine, it's normal. Pull up the various documents (requirements, specs, design, test plan). Test plan? Hah! It was only 50 or so pages. It was nowhere near comprehensive enough to verify the system behaved correctly. So we started expanding it. We set up our test stations and started flipping switches and developing new plans. Crap. Within a week we found a problem. Some background: These are safety systems. We made (for this particular project) the sensors that would be positioned in various places (think smoke or heat detectors) and the box that aggregated the results. We then sent the aggregate results to a control panel.
So I said we took input from multiple sensors. The test plan had each sensor tested individually. Trigger a smoke alarm on A, see it gets reported. Trigger a smoke alarm on B, see it gets reported. We decided to trigger multiple simultaneous alarms. I mean, it wasn't an explicit requirement but it was certainly a reasonable derived requirement.
Odd. Only the first alarm shows up. Trigger A, see A. Trigger B while A is still on, still only see A. Alright, try the other way. Shit. Same behavior, but you only see B.
Ok, fine. What happens if we clear the alarm on A and then trigger B. Oh god. B never gets reported.
This is the primary responsibility for these systems. If they aren't doing this, they have zero value. Essentially, you would only be able to deal with one emergency. And until a system reset, you would never be able to see any others.
Another guy and I raised quite a stink but we were both too young to want to risk our careers by escalating this too high. We left it about 3 levels above us (the systems engineer). Within a year he and I were both gone, that had left a bad taste in my mouth and he had been needing a change for a while anyways.
Being fairly vague because the quality of new software was actually pretty high. And from my former colleagues it's only improved since I left. But those old systems...
[0] 100% turn over the year before I got there, I did not know that going in.
I was in V&V at the time, not yet a developer (professionally). We had a maintenance task, an old system that was built some 15+ years before any of us [0] had started working for this company. Fine, it's normal. Pull up the various documents (requirements, specs, design, test plan). Test plan? Hah! It was only 50 or so pages. It was nowhere near comprehensive enough to verify the system behaved correctly. So we started expanding it. We set up our test stations and started flipping switches and developing new plans. Crap. Within a week we found a problem. Some background: These are safety systems. We made (for this particular project) the sensors that would be positioned in various places (think smoke or heat detectors) and the box that aggregated the results. We then sent the aggregate results to a control panel.
So I said we took input from multiple sensors. The test plan had each sensor tested individually. Trigger a smoke alarm on A, see it gets reported. Trigger a smoke alarm on B, see it gets reported. We decided to trigger multiple simultaneous alarms. I mean, it wasn't an explicit requirement but it was certainly a reasonable derived requirement.
Odd. Only the first alarm shows up. Trigger A, see A. Trigger B while A is still on, still only see A. Alright, try the other way. Shit. Same behavior, but you only see B.
Ok, fine. What happens if we clear the alarm on A and then trigger B. Oh god. B never gets reported.
This is the primary responsibility for these systems. If they aren't doing this, they have zero value. Essentially, you would only be able to deal with one emergency. And until a system reset, you would never be able to see any others.
Another guy and I raised quite a stink but we were both too young to want to risk our careers by escalating this too high. We left it about 3 levels above us (the systems engineer). Within a year he and I were both gone, that had left a bad taste in my mouth and he had been needing a change for a while anyways.
Being fairly vague because the quality of new software was actually pretty high. And from my former colleagues it's only improved since I left. But those old systems...
[0] 100% turn over the year before I got there, I did not know that going in.