I'm not arguing for or against that oath, but that would 100% kill Open Source dead the minute it becomes a requirement. Absolutely no one would share their code if they could be held responsible for its misuse.

Yep. How much do you want to bet that the Chinese surveillance systems that target the Uighur run on some version of Linux? I bet that they have some parts that are written Python, Java, Javascript, C or C++ for which they use an open source compiler.

> If my technology is used for evil, I am responsible.

So, by that logic, Tatu Ylönen should be held responsible for all hacks/crimes committed via ssh?

You can't blame someone for making a tool that someone else uses for evil. Should Ford be responsible for all auto-related deaths? Should Edison be responsible for all deaths where the state put someone to death via electrocution? Should Jobs be responsible because a hacker used an OSX machine to hack into another machine?

I don't get this line of thinking, although I understand the sentiment.

I don't know if the oath comparison works out so well for development. Let us not forget that each physician is directly responsible for a life every time they see someone. They also have a more direct effect on the success or failure of the procedure as they tend to produce a large portion of the work.

I'd argue few companies ever reach this level of risk, and those that do are so large that the individual contributor cannot reasonably take on that burden of responsibility.

In the example of some Amazon surveillance 'big-brother' software: Max the software dev is just making facial recognition software to the best of their ability. They aren't privy to the motivations, long-term plans, and potential consequences of those decisions.

The oath is always a fun topic to discuss though: In reality it holds no meaning other than to the one who takes the oath. Correct me if I'm wrong, but in malpractice cases I doubt they cite the oath as evidence since all students are essentially forced to recite it.

No snowflake is responsible for the avalanche, therefore I must pile on as much snow as possible -- The modern developer.

But the snowflake doesn't know that it is falling on a mountain slope.

Individual snowflakes are not capable of abstract reasoning.

Can we make it a requirement for ourselves to limit our power to our ability to keep that power safe?

I think that’s a superset of the problem of incentive alignment in AI safety, so probably not... but we also shouldn’t let the perfect to be the enemy of the improved.

The analogy works better than you think - gravity is responsible for the avalanche. They are all responding to irresistible forces.

No individual litterer is responsible for a trashed park.

> I don't know if the oath comparison works out so well for development. Let us not forget that each physician is directly responsible for a life every time they see someone

I commit every single day of my professional life as a physician to do my best for my patients, and to do the absolute least harm possible. And over the entire course of my lifetime, I will not achieve the scale of harm - or benefit - that a developer can achieve with a few months or years of concentrated effort.

That is insanely broad and vague. You can be responsible for knowingly designing a system for abuse, or if you give it to someone you know, or have reasonable cause to suspect, will abuse it. But what you're saying here is that if you ship anything at all, and someone somewhere finds a way to use it for some nefarious purpose, then you're responsible. That's not how things work with anything else; why should they work that way with software?

That aside, in US specifically, there's already something that is more narrowly tailored to our current reality. They aren't accepting new signatures because of how many there were, but you can still make the same pledge (and e.g. share it publicly to have some skin in the game):

http://neveragain.tech/

"Anything I build can and will be abused... If my technology is used for evil, I am responsible."

Well, which is it? Both of these statements can't be true. Does this creed apply to gun manufacturers? Knife manufacturers? Car manufacturers? Hammer manufacturers?

If this effort succeeds, the end result will be something like "know your customer" for cloud providers. So, just like with app stores, you'll have to worry about getting your service approved.

There seems to be an eternal cycle of abstraction creating and breaking, in both finance and software.

- Someone builds a general-purpose abstraction that becomes popular.

- Inevitably, the abstraction gets abused.

- To prevent abuse, the abstraction gets violated. It's not general-purpose anymore because some usages are disallowed.

(This seems to be why open API's to online services tend not to last very long.)

It depends on how far down the line you want moral culpability to apply. To take this to it’s logical extreme, it effectively makes the developer responsible for anything the end user decides to use their software for. The same reasoning is not only not applied to software, it is not applied to any product class with the exception of those that are immediately and obviously lethal. Engineers in google would be responsible for drug cartels operating in Columbia.

Higher order causation must be separated from moral culpability. In other words, you should not hold people responsible for things that happen far downstream. Things that happen several links down the causal chain have occurred due to the decisions of many other people further along that chain, and a higher culpability should fall upon them.

That is not to say that a software developer cannot be directly responsible for bad outcomes, maybe for example working on weapon systems for a nefarious state, where you’re fairly close causally to the point of application. My point is that it’s not a good idea to push this to it’s limits.

Has anyone actually quit over this? http://neveragain.tech/

So, if I'm an assembly line worker at GM and someone buys a pickup truck and intentionally drives it into a crowd, murdering 20 people, I should get the death penalty as a serial killer?

Obviously hyperbolic, but not very different at the core.

A car’s primary function is not to kill people. If you are an engineer and develop a tool whose primary purpose is mass surveillance, you should bear that ethical burden.

And yet, cars kill many, many more people (1.25 million in 2013, per WHO) than mass surveillance does. And parent didn't qualify his statement as to which products the oath would apply to:

"Anything I build can and will be abused. I am responsible for my designs, for my products, and for the data I collect and store. If my technology is used for evil, I am responsible."

Mass surveillance does something a lot more insidious than car wrecks, it sets up a future where people are killed. It's like global warming.

Whatever the "primary function," car accidents is the leading cause of young people's death in the US.

I think oaths are completely outmoded as a social construct - everyone knows that people lie regardless of what they claim. The actual teeth to it were their own sources of problems, guilds and not being able to run your business if violated. They were supplanted by licenses.

Even if we somehow had that measure in place it would be a cure worse than the disease - well you can't get a job doing something legitimate because it turns out that your safety human recognition algorithm got abused by somebody else? Just crime for you to make a living.

I think it's time to acknowledge the impact of culture in our oaths. It IS how we're all naively programming ourselves and others, so we might as well address it.

If my tech is used to do harm, the responsibility to heal is shared between myself, the cultures involved in leading to harm, the perpetrators of the harm, those harmed, those witnessing harm, and those in denial of responsibility. I'll do my part while encouraging and believing in others' willingness to grow together.

Worth noting the Project Managers who are certified "PMP"s attest to a code of conduct that includes: "take actions based on the best interests of society, public safety, and the environment." It requires that PMP's report any unethical practices to "appropriate management". Pretty weak, but hey, it's better than nothing.

Something like this would be a great start. I have a feeling the industry has a tendency to try to recruit young and perhaps naive developers who might not be fully ready to appreciate all of the potential misuses (and the probability of those outcomes) of what they build, however. Or worse, have some misaligned incentives to build them regardless.

I like the idea of improving a sense of developer responsibility. But the challenge here is how to you define evil? How would you interpret this oath when building a web browser? Have a blacklist of "evil" sites? Obviously, a lot of issues to sort through there.

So if you write an encrypted chat app, and people use it to plan a crime, are you responsible?

What is "evil"?

It's almost as if workers should own and control the means of production.

There is absolutely nothing stopping you and your friends from starting such a company.

I won't be holding my breath as you start on your new venture.

Personal attacks will get you banned on HN. Please don't post like this again.

https://news.ycombinator.com/newsguidelines.html

Nothing holding anyone back except pesky things like paying rent and buying food.

We do.

In software, at least.

The world has moved on a lot since 1848 — someone needs to promote genuinely new political ideas for our era, not rehash ones that came from the transition from agrarian-feudalism to industrial-capitalism.

"Anything I build can and will be abused. I am not responsible for my designs, for my products, and for the data I collect and store. If my technology is used for evil, the corporation employing me is responsible and the government is at fault for not doing something about it."

Yes, you are. You chose to follow the design and build a system you knew would be abused. This mindset is what got us here to begin with. YOU ARE AT FAULT.

Agreed. The USA is all about individual liberty, but not responsibility, it would appear.