If you write software that replaces someone's job are you "culpable"? If you create a phone and people are killed when someone texts and drives are you "culpable"? If you create a better facial recognition program as part of building a secure access systems and someone uses that system to discriminate are you "culpable"?
I find this unending process of creating "victims" just so they can be "saved" as tiresome.
If a thief comes and steals money under the mattress meant to pay for your kid's college tuition, and your kid is dropped because she can no longer pay, your kid's loss is a second order effect of the thief's theft, but I still hold the thief responsible*
* this is a hypothetical story; please don't save that kind of money under your mattress.
1. Did they have nefarious aims?
2. Did they reasonably attempt to consider and solve issues surrounding first and second-order effects of their creation?
3. How did they react to previously unknown first and second-order harms when they found out about them?
The first two, I tend to give creators the benefit of the doubt about. But I've become quite dismayed by the responses of creators to the harms of their creations.
Tell me, in the VW emissions scandal, did that happen ? No.
Germany literally changed the laws retroactively to avoid holding their own companies to account. This is technically illegal, but the judicial system REFUSED TO LOOK AT IT.
Given that both management of these firms, and government wants to commit fraud when culpability is assigned ... what point is there discussing the rules ? They won't be applied when it matters.
But of course there are large advantages to people believing they will be applied. Like there are advantages to people not responding to having their wallets taken from their pockets, their children kidnapped for sale into abuse, their houses robbed empty ...
This is the US government in action. Other governments, including European, are worse, not better. These are the people that make, and enforce the rules. Why are we discussing whether the right things will be happening ?
It is the second word, intent, that is the hang up. If a person hurts someone but it was not their intent to do so, that person is not typically to be held morally responsible. So, at least in the western concept of moral culpability, intent matters. But... people also have a responsibility, a responsibility to try to anticipate the consequences of their actions. One cannot shield oneself from culpability by intentionally remaining ignorant. I think that engineers avoid thinking about these things, not out of humility, but because its in their interest to do so.
Absolutely agree.
> I think that engineers avoid thinking about these things, not out of humility, but because its in their interest to do so.
This is perhaps the reason we differ. My experience has been that the vast majority of technologists want to make the world a better place.
I believe you see the industry is filled more with people like Wernher von Braun who want to build something cool regardless of social cost. At the dawn of the social media revolution the dominate theory in tech circles was that information shall set you free. The people making it did not come into the industry for status, being a nerd was still a social negative. Today with the ever rising salaries things are changing.
Perhaps you are right that as the industry matures the sphere of liability should increase. I would still argue that the suggestions in the original article swing that much too far. We shouldn't prevent scientific discoveries merely because engineers may build something bad with the information. On the balance scientific discovery has improved the lives of everyone by many orders of magnitude - even if its distributed unevenly.
Again a first order effect
> If you write software for autonomous cars that is not foolproof at detecting pedestrians, you are culpable.
Again a first order effect connected directly with the immediate goal.
However if you write communications software and people use it to say bad things, those are second order effects. You provided communication software that works as advertised and has beneficial purposes. Others misused it.
Not trying to be inflammatory. This is specifically trying to find if you think there's a point where culpability disappears. I would appreciate a yes/no answer. There's no nuance to the question. I'm not talking about the specific valves, just the notion of a valve.
Did you mean first order or second order kind of like the way physics people mean it (direct linear relationship with strong effect on the unknown variable). Or the way that MBAs mean it (they have their own definition).
Or did you mean structural engineering, same as physics: https://www.quora.com/In-structural-engineering-what-are-fir...
I think what you probably mean is a colloquial interpretation, where first order means something is directly causal in an obvious to see way, where second order is more of an indirect effect that occurs through complex dynamics.
Either way, I don't think those terms are well defined, and they're kind of complex enough that using them to argue for morality, and calling people out for not knowing your definitions, makes you look caddish.
I doubt that they could mean causally indirect. I think it must mean something like: not used in accordance to their the engineer's intent, especially when that intent covers a a general affordance (e.g. affords communication over long distances), but is being judged for a more specific affordance (affords communication over long distances to order a terror attack). I dunno. I'd have to be a philosopher to work it out.
The problem of where to draw those lines is so difficult that legal systems resort to constructing tests based on whether a "reasonable person" could have foreseen the specific consequence of an act.
If you write software for autonomous cars that is not foolproof at detecting pedestrians, you are culpable. In aviation, catastrophic failure means at least one death and the system should have an overall probability of failure at 10^-9. Critical software is frequently written at one or two orders below. That means 95%, 99%, 99.9%, are not enough.