'Many forms of Government have been tried, and will be tried in this world of sin and woe. No one pretends that democracy is perfect or all-wise. Indeed it has been said that democracy is the worst form of Government except for all those other forms that have been tried from time to time.'
https://richardlangworth.com/worst-form-of-government
(OK, that was slightly OT!)
Obviously we need constitutional rights and bureaucrats to implement the will of the people as expressed in the polls. But why do we need representatives? Do you really think eg a Tea Party representative will protect Muslims from the majority better than the first amendment and courts?
No. The lesson, as usual, is that extremes rarely work out for us. The plutocrats will screw us if totally in control. So will politicians if they can get money from them or special interest groups. So will democracy if something horrible is trending in popularity. As usual, the solution will be a compromise between various points in the design space with more effective checks on various interactions and risks each pose to each other. I can't tell you what that design will be but looking for it is best investment with these things.
"Do you really think eg a Tea Party representative will protect Muslims from the majority better than the first amendment and courts?"
The First Amendment is subject to interpretation by representatives and/or courts. Fourth, Fifth, and Fourteenth have seriously been watered down. They're among the best examples of the risk. A politician can outlast the media fervor of the moment, get advise from well-informed people, and have teams check the side effects of the law. The current situation exists because apathetic democracy allows them to get away with not doing that. Also, passing laws in exchange for bribes. That could all be stopped with enough voter action followed by legislation or alternative models. The benefits of representational democracy remain.
I'm not saying I believe it's the best system. I'm just saying plutocrats and mobs cause lots of problems it can prevent with less effort than constantly outsmarting plutocrats or fighting mobs. So, it's a consideration.
Or it exists because it is an emergent phenomenon that arises eventually. An increasingly polarized two-party system where each primary race is rigged against outsiders and voters are scared to vote third-party because each candidate is super scary to the opposing party. It's bound to happen eventually. In this case it happened in the US presidential race. And Congress has long had a 10% approval rating. One can make up reasons as to why it happened or accept that a representative democracy eventually reaches such states, and it's not clear how to get out of them with more representative democracy. Trying the same thing and expecting a different result.
I'm not saying I believe it's the best system.
Exactly, and I'm saying it's not. It's easier to fool all of the people some of the time (eg during elections) or heavily influence some the people all of the time (as lobbyists for special interests do) but you can't fool all the people all the time. That is far more expensive.
Once again ask yourself: are eg Muslim and Mexican US citizens safer if Donald Trump gets elected and magnifies the desire of those who elected him, or if the population at large was polled for policy? Are the people of Gaza better off under Hamas because they were democratically elected once by those who showed up? Would the Nazis ever get into power if there were no representative positions to get into? Besides gerrymandering, poor voter turnout and other major problems, representative democracy eventually gives a giant hammer to some special interest or other.
The crowd would do a better job on predicting policy outcomes than experts: http://www.npr.org/sections/parallels/2014/04/02/297839429/-...
I can't proof that the neutral network always behaves correctly, but with testing and statistics you can verify the network up to a chosen certainty (below 100% obviously). Or you can actively search for problems with fuzzing techniques. All without solving the halting problem.
So you'll end up with ANN's reviewing other ANN's which are reviewed by other ANN's at the end.
You can test ANN's by simply fuzzing them and seeing the output, getting much insight to why an outcome was chosen for a given data set is near impossible.
https://people.cs.umass.edu/~btaylor/publications/PSI000008....
These are also just the first steps. Not bad given what hardware and software verification used to look like a few decades ago. I think the verifiable ANN's will be more structured or constrained in their development or even use cases.
"it's simply beyond the human capability to review. So you'll end up with ANN's reviewing other ANN's which are reviewed by other ANN's at the end."
That implied it was beyond human capability to review ANN's to the point we'd need ANN's to do it. In the paper, they give a number of deterministic methods to review ANN's noting some were successful by the teams using them. Countered both of your claims in that context.
"This means that having the ANN being open sourced does not add additional security by having the code inspectable by everyone like with traditional software."
I agree that most of the problems won't be caught here. More about training set and safety monitors. However, the paper shows methods like rule extraction, visualization, and simulation that have aided in assessing and improving neural networks. These work on that internal state that's incomprehinsible at first glance. It's also usually stored in HW logic, the SW code, or some data format. Having those provably facilitates analysis of ANN's performance. Assuming it's the kind that's analyzable at all. ;)
Note: Code-level errors, either by hand-written or auto-generated engines, can also invalidate an ANN's behavior. Vast majority of software flaws occur right here. Obviously, some static analysis or safety transformations can increase assurance of correctness of overall system. Should be combined with other V&V methods.
If you get a the source code of any traditional software reviewing the code can be done by humans regardless of how complex it is, with ANN's it's not really the case simply because of the nondeterministic nature of the system. You can validate the "correctness" of an ANN but it's not done in the same manner as reviewing traditional software.
You have to do both: validate the non-deterministic aspects of the ANN logic and validate its implementation as code. Makes them a lot harder. One reason I avoid them.
So, not only can you prove a program halts using a verified halter, the techniques for doing so have prevented and detected many real-world problems. One of those situations where abstract math ignores simple, concrete solutions. :)
Safety critical automotive software is going to tend to be an easier case, since it's quite likely to be extremely time bounded.
You can, actually, the result would be extremely complex, though.
You're ignoring the biggest issue in your whole tangent: real-world engineers with sense won't use NN's in systems unless the problem itself is non-deterministic & other methods don't work. Personally, I push control theory, state machines, ladder logic, whatever for applications like this. You can deterministically analyze them. Whereas, problems with too much uncertainty or requiring approximate solutions will always have an error margin. You personally have been pushing this thread to focus as much as possible on the NN case while ignoring 99% of real-world, use cases representing how the OP topic will usually play out. You also act like the inherent error in the tooling is bad when the problem domain itself has inherent error. So, I decided to address your tangent.
The first solution, safety monitors, goes way back in safety-critical use. That's to have a simple solution checking the ranges or other behavior of the NN that's able to override faulty NN with simpler logic. This can be done for GPS-guided drones that usually move in straight lines or smooth curves. The NN part is reset, diagnosed, or deactivated while the fail-safe system takes plane on simpler path or out of the area.
Second part checks the nets themselves. Again, given a non-deterministic problem domain, the nets will be non-deterministic with the best one can do getting the error margin way down or within fail-safe bounds. This other paper I found summarizes a lot of V&V methods for NN's:
https://people.cs.umass.edu/~btaylor/publications/PSI000008....
Safety monitors. Training data or updates with constraints built in. Automated, test-case generation tweaking expected data. Rule extraction. System visualization. Simulation. All should contribute to safer NN's. They can't be perfectly correct, though, because vision, complex flying, thinking... none of this is 100% accurate no matter what method you use.
The decision itself might be taken using an amount of data completely overwhelming to human being analysis, processed at inhuman speed.
But even in safety critical tasks I have seen NNs and MDPs used to derive a good enough solution, then training turned off and the system certified (e.g. by running a large gold standard collection of test cases)
I broke it down here with a quick revision just now:
And good luck fixing heartbleed in a proprietary product in the way that occurred for OpenSSL - the entire codebase got looked at and we got BoringSSL and LibreSSL.
What would you prefer - code with obscure bug from spaghetti code so bad it kills people [1], or code looked at by enthusiastic amateurs and professionals alike? Car people tend to enjoy pushing the envelope and they are doing this with software already!
1. http://www.safetyresearch.net/blog/articles/toyota-unintende...
https://www.fsf.org/blogs/community/who-actually-reads-the-c...
On the flipside, if few use your project, bugs can last for ages. I've had this experience also- distributing projects with free source that don't work, and not finding out until much later when I noticed it myself.
Making source free is not bad for security, because it's not about high visibility. Security problems are more frequent when the vector of attack is larger, e.g. when your car can be stolen because they didn't anticipate misuse of the entertainment system that was accessible via the carrier.
Related to that topic, Chrysler has recently been paying hackers to try to find security flaws: https://www.wired.com/2016/07/chrysler-launches-detroits-fir...
If the source is free or open, then people can find flaws more easily, so that they can be fixed. You want to be able to find bugs fast and fix them quickly. When you don't, that's when you get into serious trouble.
As a warning though, source being free and open and well-used doesn't mean that bugs can't go a very long time without being seen, after which point they are everywhere: https://en.wikipedia.org/wiki/Shellshock_(software_bug)
What are the bad effects to society for software transparency? Because if open software offers noisy positives with minor negatives, then we should prefer open as the default heuristic until circumstantial information says otherwise.
Some projects, whether open or closed, are more valued by institutions or businesses, and consequently some projects will justify more money and eyeballs. Also, what businesses and institutions value can be unintuitive. When there's a bug in Linux, that might immediately degrade a business product, whereas a security problem with SSH might not ever impact a business -- even if important customer data is stolen. I doubt Target experienced a real business problem after its large data theft incident with customer financial data.
Therefore, 3rd parties contributing to software will probably be far more prompt for bugs in projects that directly impact business success vs. OpenSSH, but at least the door for extra benefits is open, whereas for closed software the door is shut.
What are the bad effects to society for software transparency?
One is companies that are less willing to write code or go to market because their code IP wouldn't be competitive given existing players. Smaller new competitors' code would be quickly copied by larger existing players with their larger teams into larger projects that are already used by everyone.
Then there is the case of artistic one-off projects. Imagine a $50 video game releasing its source code immediately or as it is being made. There would be another company that just copies the code or the textures and distributes quickly before the last level is done.
at least the door for extra benefits is open, whereas for closed software the door is shut
This is true, but some doors that you don't want to be open are also shut. It takes a lot of effort to reverse engineer something. I bet all the openssl bugs would be harder to find if it wasn't open source. Especially, if binaries weren't available either - something like only an API layer existing. Maybe heartbleed wouldn't have been found until openssl was replaced by something else.
Finally, something like GNU parallel is very commonly used. On the other hand, some random utility that is on Github and isn't very popular probably isn't looked at by nearly as many people. Out of 100 people who use some software, about 1% looks at source code. So then if you have 1000 people use your tool, only 10 people look at the tool's source code and 1 of them is a malicious actor looking for exploits. I have personally found several bugs in open source software that could have been security exploits, but haven't reported them because I was lazy at the time.
Having said all this, most non-application-specific software could be open source and it would probably help it more than hurt it.
~ artists will create fewer video games
To me these are arguments for making closed source illegal, especially the second. Among many benefits of first, fewer actors warping society in order to capture rents from software they control, eg tax software vendors. On second, destroy one incentive to create addicts.
Maybe not with the software most of the world doesn't care about but I'd bet that there will always be someone willing to look real hard at software being put in to popular automobiles.
Reminds me of the common confounding of elections and democracy.
Try again, that is bsd openssl code, probably proprietarized, not gpl.
Well, no. No it isn't. A higher possibility of more people seeing the code translates into a higher probability of bugs getting fixed. Of course, there's no guarantees, but more eyes means more chances of bugs getting caught. And that means safer.
That doesn't. It means potential to be safer if all kinds of things line up. Vast majority of FOSS code is crap like any other code. More eyes didn't change that. What's needed is design/implementation work, qualified review, time, and trustworthiness of reviewers & distribution.
1. Rigorously designed & reviewed open-source is greater than...
2. Rigorously designed & reviewed closed-source is greater than...
3. Typical, open-source is greater than...
4. Typical, closed-source is greater than...
5. Cloud infrastructure based on Concurrent DOS written in a MOL-360 language.
The idea that software is inherently safer because it's released under an open-source license is overly simplistic in the extreme. Just the most obvious reason for this is that opportunity for independent review doesn't guarantee that review will happen. The wisdom of the crowd doesn't guarantee a third party review will be better or more thorough than a solid first party system. Open-source provides the possibility of review, and that's all. Hypothetical eyes make no bugs shallow.
Edit - I'm not making a claim one way or the other about the relative safety of open vs proprietary software, I'm merely saying that the comparison is deeper and more nuanced than the way it is typically portrayed.
1 - http://www.bbc.com/news/technology-28867113