The thing I've never understood about philosophy is that why are ethics and good/evil considered to be properties of people as opposed to systems? It seems like all the social science research points to the conclusion that the intrinsic nature of a person, if such a thing even exists, has almost nothing to do with their behavior.

So you would think that there would be dozens of papers that analyzed the trolley problem from the point of view of what makes an algorithm ethical or unethical, but this doesn't seem to be the case. (E.g. why not imagine a universe where the people on each track had to somehow bid via differing auction mechanisms as to which direction the train went, why even imagine a person at the switch at all?)

I think you're looking for decision theory. http://en.wikipedia.org/wiki/Decision_theory

What I'm interested in though isn't what makes a decision by a person ethical or rational or whatever, but rather what makes a social system itself (or an algorithm that interfaces with people) ethical or unethical. E.g. looking at the ethics of the design the social system behind the Zimbardo prison experiments, rather than looking at the decisions of the individual actors once they're already in that environment.

I am somewhat interested in questions like whether or not it's morally acceptable to lie to a system or a (non-conscious) robot. At least on the surface this doesn't violate any of Kant's moral objections to deception. That is, Kant said (more or less) that the reason why lying is unethical is that it deprives the other person of their ability to make a rational decision, and our rationality is part of what makes us human, so lying is unethical because it deprives others of their humanity. So e.g. is it unethical to lie to about your age on a porn site? It might be, but if it is then we'd an entirely new framework to justify that sort of thing.

It almost seems like based on the power imbalances between people and algorithms, and the traditional justification against lying, it would almost be morally virtuous to teach your children to lie, as long as they're lying to a machine and not a person. (Of course a machine could be composed of people who are following an algorithm when interacting with you, but they still aren't making any sort of autonomous decisions for themselves.)

Anyway this type of problem is a little different than the problem of what makes the systemic design of a social system ethical or unethical, but they're both just different sides of the same coin.

you can evaluate the decision function of groups of people. coming up with ways to morally justify defrauding or committing violence against groups of people by abstracting away their humanity has been a favorite pastime for millennia.

I don't keep up with sociology/ethics literature but I haven't heard your original thesis before and its an interesting discussion.

I'm sure there were otherwise nice people that were Nazi soldiers. And there are probably a few nasty people who work for Google to create products millions of people love.

As social systems, as you call them, grow larger and more connected it may make the ethical positions of systems more important. The problem is that most corporations are inherently amoral, and governments almost always think they are the good guys.

I've always thought that systems were inherently evil. By goerdel's theorem, you won't be able to exhaustive list enough rules to make the system perfect. Therefore, we always must rely on individuals making human moral judgements. (This is a way simplified personal philosophy that I've never seen anyone articulate well. Maybe I'll have to write a treatise on of someday.)

Great point. Although the YC universe has plenty of obedience-to-authority elements as well -- think of how many times people on HN invoke "the rules" to shut down discussions about the TSA.

The naughtiness stuff works on HN despite this, probably since you can't downvote articles and an article eventually getting flagged into deep space doesn't negate the upvotes it received beforehand. The system relies on humans not rewarding intensely but shallowly interesting articles, but perversely these remain a quick route to points.

Nb. I think that recent discussions have put a moderate damper on this, and perhaps the newer ranking algo (in which flagging has greater weight I think?) has played a part, but the problem remains.

On the other hand, Hitler, Stalin, Mao, etc, broke a few rules too. So I am not sure that championing rule breaking is going to help improve society.

More likely, what's actually needed is a balance. And, interestingly, the kind of person that makes a good entrepreneur is probably on the psycopathic side of normal. A very nice article in the Guardian made this point at the weekend - http://www.guardian.co.uk/books/2011/may/21/jon-ronson-how-t...

Did you honestly understand my comment as implying that rule breaking automatically makes you a good person? (regardless of which rules are broken)

Conversely, breaking all the rules almost always makes you a bad person.

Nobody can ever break all the rules - the laws of physics might have something to say about it.

If you limit your statement to "all the rules of the society in which you live", I'm not sure your statement is true. What do you think of Oskar Schindler? Womanizer, cheat, Nazi, fraud, and traitor. Yet he saved the lives of hundreds of Jews during the holocaust, and ended up as the hero of a movie.

The way I work this out is to consider that there are rules, and principles. Breaking some rules is ok, as long as principles are solid behind. Moral principles tell me I should not betray my friends or my family, for example. If the issue is whether or not I should game SEO, lie to my boss or scan competitor servers for vulnerabilities, it is more on the "rules" side.

If I break a rule, I still can stare at myself in the mirror, and I will even be able to tell this to my children when they will be grown-ups. If I happen to break one of my moral principles, it will be another story.

This is nothing new, Confucius taught this long time ago already.