News for hackers. Stuff that matters.
Re:Software Freedom Doesn't Kill People, Security Through Ob... by ktRolster on Sunday August 14, 2016 @04:18AM (#12284752)
With open source software, you have the possibility of knowing how safe it is. With closed source, you don't (ok, it's a lot harder to read through binaries, but still vaguely possible).
Re:Software Freedom Doesn't Kill People, Security Through Ob... by tigershark on Sunday August 14, 2016 @05:01AM (#12284885)
No way that you can understand how safe is the neural network that is driving your car. And this applies equally to open source autopilot systems and closed ones. The article is just a rant that misses this fundamental point.
Re:Software Freedom Doesn't Kill People, Security Through Ob... by amalcon on Sunday August 14, 2016 @09:05AM (#12285420)
Sure you can. It's just not by verifying the algorithm, it's by analyzing real-world performance. This is the same way that we understand how safe the human driving the car is, and the insurance industry is very good at this.
Re:Software Freedom Doesn't Kill People, Security Through Ob... by eridius on Sunday August 14, 2016 @03:27PM (#12286924)
Analyzing real-world performance doesn't care if the source is open or proprietary, so you're kind of missing the point of this whole thread.
Re:Software Freedom Doesn't Kill People, Security Through Ob... by chmars on Sunday August 14, 2016 @05:06AM (#12284904)
I cannot, others can - I can even pay them to do if necessary.
Re:Software Freedom Doesn't Kill People, Security Through Ob... by EGreg on Sunday August 14, 2016 @08:43AM (#12285366)
Can you pay someone to solve the halting problem?
Re:Software Freedom Doesn't Kill People, Security Through Ob... by chmars on Sunday August 14, 2016 @09:44AM (#12285535)
I think we all agree that open source is not the holy grail but it shines with comparative advantages. It is like democracy: Far from perfect but still the best political system:

'Many forms of Government have been tried, and will be tried in this world of sin and woe. No one pretends that democracy is perfect or all-wise. Indeed it has been said that democracy is the worst form of Government except for all those other forms that have been tried from time to time.'

https://richardlangworth.com/worst-form-of-government

(OK, that was slightly OT!)

Re:Software Freedom Doesn't Kill People, Security Through Ob... by EGreg on Sunday August 14, 2016 @10:38AM (#12285729)
Re:Software Freedom Doesn't Kill People, Security Through Ob... by nickpsecurity on Sunday August 14, 2016 @07:30PM (#12287799)
Re:Software Freedom Doesn't Kill People, Security Through Ob... by EGreg on Sunday August 14, 2016 @08:23PM (#12287945)
So the plutocrats will protect us from the majority? :)

Obviously we need constitutional rights and bureaucrats to implement the will of the people as expressed in the polls. But why do we need representatives? Do you really think eg a Tea Party representative will protect Muslims from the majority better than the first amendment and courts?

Re:Software Freedom Doesn't Kill People, Security Through Ob... by nickpsecurity on Sunday August 14, 2016 @09:20PM (#12288087)
"So the plutocrats will protect us from the majority? :)"

No. The lesson, as usual, is that extremes rarely work out for us. The plutocrats will screw us if totally in control. So will politicians if they can get money from them or special interest groups. So will democracy if something horrible is trending in popularity. As usual, the solution will be a compromise between various points in the design space with more effective checks on various interactions and risks each pose to each other. I can't tell you what that design will be but looking for it is best investment with these things.

"Do you really think eg a Tea Party representative will protect Muslims from the majority better than the first amendment and courts?"

The First Amendment is subject to interpretation by representatives and/or courts. Fourth, Fifth, and Fourteenth have seriously been watered down. They're among the best examples of the risk. A politician can outlast the media fervor of the moment, get advise from well-informed people, and have teams check the side effects of the law. The current situation exists because apathetic democracy allows them to get away with not doing that. Also, passing laws in exchange for bribes. That could all be stopped with enough voter action followed by legislation or alternative models. The benefits of representational democracy remain.

I'm not saying I believe it's the best system. I'm just saying plutocrats and mobs cause lots of problems it can prevent with less effort than constantly outsmarting plutocrats or fighting mobs. So, it's a consideration.

Re:Software Freedom Doesn't Kill People, Security Through Ob... by EGreg on Sunday August 14, 2016 @10:22PM (#12288271)
The current situation exists because apathetic democracy allows them to get away with not doing that. Also, passing laws in exchange for bribes.

Or it exists because it is an emergent phenomenon that arises eventually. An increasingly polarized two-party system where each primary race is rigged against outsiders and voters are scared to vote third-party because each candidate is super scary to the opposing party. It's bound to happen eventually. In this case it happened in the US presidential race. And Congress has long had a 10% approval rating. One can make up reasons as to why it happened or accept that a representative democracy eventually reaches such states, and it's not clear how to get out of them with more representative democracy. Trying the same thing and expecting a different result.

I'm not saying I believe it's the best system.

Exactly, and I'm saying it's not. It's easier to fool all of the people some of the time (eg during elections) or heavily influence some the people all of the time (as lobbyists for special interests do) but you can't fool all the people all the time. That is far more expensive.

Once again ask yourself: are eg Muslim and Mexican US citizens safer if Donald Trump gets elected and magnifies the desire of those who elected him, or if the population at large was polled for policy? Are the people of Gaza better off under Hamas because they were democratically elected once by those who showed up? Would the Nazis ever get into power if there were no representative positions to get into? Besides gerrymandering, poor voter turnout and other major problems, representative democracy eventually gives a giant hammer to some special interest or other.

The crowd would do a better job on predicting policy outcomes than experts: http://www.npr.org/sections/parallels/2014/04/02/297839429/-...

Re:Software Freedom Doesn't Kill People, Security Through Ob... by nickpsecurity on Monday August 15, 2016 @01:06PM (#12291728)
I'm going to have to think on these points some more. :)
Re:Software Freedom Doesn't Kill People, Security Through Ob... by halomru on Sunday August 14, 2016 @09:09AM (#12285436)
No, but neither can anyone else.

I can't proof that the neutral network always behaves correctly, but with testing and statistics you can verify the network up to a chosen certainty (below 100% obviously). Or you can actively search for problems with fuzzing techniques. All without solving the halting problem.

Re:Software Freedom Doesn't Kill People, Security Through Ob... by EGreg on Sunday August 14, 2016 @09:12AM (#12285444)
Right, but with the code in the open, it's easier to find ONE easy hole than to patch ALL the easy holes.
Re:Software Freedom Doesn't Kill People, Security Through Ob... by dogma1138 on Sunday August 14, 2016 @04:42PM (#12287244)
No that's not really how ANN's work.
Re:Software Freedom Doesn't Kill People, Security Through Ob... by EGreg on Sunday August 14, 2016 @06:24PM (#12287623)
ANNs?
Re:Software Freedom Doesn't Kill People, Security Through Ob... by dogma1138 on Sunday August 14, 2016 @06:31PM (#12287641)
artificial neural networks, you won't really be able to opening the code won't tell you much if anything, it's simply beyond the human capability to review.

So you'll end up with ANN's reviewing other ANN's which are reviewed by other ANN's at the end.

You can test ANN's by simply fuzzing them and seeing the output, getting much insight to why an outcome was chosen for a given data set is near impossible.

Re:Software Freedom Doesn't Kill People, Security Through Ob... by nickpsecurity on Sunday August 14, 2016 @07:32PM (#12287803)
Lots of ways to review them:

https://people.cs.umass.edu/~btaylor/publications/PSI000008....

These are also just the first steps. Not bad given what hardware and software verification used to look like a few decades ago. I think the verifiable ANN's will be more structured or constrained in their development or even use cases.

Re:Software Freedom Doesn't Kill People, Security Through Ob... by dogma1138 on Sunday August 14, 2016 @07:53PM (#12287867)
Have you actually read what the paper you've linked? I haven't said that it's impossible to review ANN's, but you do not need to review their code to do it, infact reviewing their code is the wrong approach in this case. You do not want to be conducting code inspection on nondeterministic systems as it would yield very little. This means that having the ANN being open sourced does not add additional security by having the code inspectable by everyone like with traditional software. This is what the whole argument is about, not about can ANN's be reviewed.
Re:Software Freedom Doesn't Kill People, Security Through Ob... by nickpsecurity on Sunday August 14, 2016 @08:00PM (#12287892)
You said:

"it's simply beyond the human capability to review. So you'll end up with ANN's reviewing other ANN's which are reviewed by other ANN's at the end."

That implied it was beyond human capability to review ANN's to the point we'd need ANN's to do it. In the paper, they give a number of deterministic methods to review ANN's noting some were successful by the teams using them. Countered both of your claims in that context.

"This means that having the ANN being open sourced does not add additional security by having the code inspectable by everyone like with traditional software."

I agree that most of the problems won't be caught here. More about training set and safety monitors. However, the paper shows methods like rule extraction, visualization, and simulation that have aided in assessing and improving neural networks. These work on that internal state that's incomprehinsible at first glance. It's also usually stored in HW logic, the SW code, or some data format. Having those provably facilitates analysis of ANN's performance. Assuming it's the kind that's analyzable at all. ;)

Note: Code-level errors, either by hand-written or auto-generated engines, can also invalidate an ANN's behavior. Vast majority of software flaws occur right here. Obviously, some static analysis or safety transformations can increase assurance of correctness of overall system. Should be combined with other V&V methods.

Re:Software Freedom Doesn't Kill People, Security Through Ob... by dogma1138 on Sunday August 14, 2016 @08:08PM (#12287906)
The full sentence was "...opening the code won't tell you much if anything, it's simply beyond the human capability to review." It was within a comment subthread about the value of traditional code inspections for reviewing ANN's more specifically will the availability of the code for everyone to inspect be as "valuable" for security assurance as traditional software, context is everything.

If you get a the source code of any traditional software reviewing the code can be done by humans regardless of how complex it is, with ANN's it's not really the case simply because of the nondeterministic nature of the system. You can validate the "correctness" of an ANN but it's not done in the same manner as reviewing traditional software.

Re:Software Freedom Doesn't Kill People, Security Through Ob... by nickpsecurity on Sunday August 14, 2016 @09:14PM (#12288067)
"with ANN's it's not really the case simply because of the nondeterministic nature of the system. You can validate the "correctness" of an ANN but it's not done in the same manner as reviewing traditional software."

You have to do both: validate the non-deterministic aspects of the ANN logic and validate its implementation as code. Makes them a lot harder. One reason I avoid them.

Re:Software Freedom Doesn't Kill People, Security Through Ob... by nickpsecurity on Sunday August 14, 2016 @07:19PM (#12287775)
I made money doing it one time. They told me I couldn't prove an arbitrary program would halt. I wrote a program that loaded an arbitrary algorithm into a process, counted down to zero from starting constant, attempted to kill the process when at zero, and otherwise shut down the whole system. "sklogic" similarly showed how pointless the question was when he said he could just put a while loop around any algorithm doing basically the same thing within the program.

So, not only can you prove a program halts using a verified halter, the techniques for doing so have prevented and detected many real-world problems. One of those situations where abstract math ignores simple, concrete solutions. :)

Re:Software Freedom Doesn't Kill People, Security Through Ob... by dllthomas on Sunday August 14, 2016 @02:01PM (#12286583)
No, but for any given program I may be able to hire someone to tell me whether it halts.

Safety critical automotive software is going to tend to be an easier case, since it's quite likely to be extremely time bounded.

Re:Software Freedom Doesn't Kill People, Security Through Ob... by tigershark on Sunday August 14, 2016 @05:12AM (#12284928)
No one can. If anyone could certify the neural network entire response mapping then you could simply write an exact algorithm with the same responses. Good luck for doing it to automatically drive a car. Strangely every company in the world is focusing on NN for autopilot, not on hand written algorithms.
Re:Software Freedom Doesn't Kill People, Security Through Ob... by ktRolster on Sunday August 14, 2016 @01:20PM (#12286396)
If anyone could certify the neural network entire response mapping then you could simply write an exact algorithm with the same responses.

You can, actually, the result would be extremely complex, though.

Re:Software Freedom Doesn't Kill People, Security Through Ob... by nickpsecurity on Sunday August 14, 2016 @07:02AM (#12285137)
Nice strawman given most systems use deterministic methods for that reason. Plus, simplest is often best in safety-critical. That said, you could've gotten more answers on Google with terms formal verification neural nets. NASA paper with good methods was first result for me.
Re:Software Freedom Doesn't Kill People, Security Through Ob... by tigershark on Sunday August 14, 2016 @07:36AM (#12285195)
Good methods for what? Not certainly to certify that each combination of inputs gives a certain output, as you can read from that paper. And that is the only way that would have ensured that the guy on the tesla didn't die. That methodology furthermore has nothing to do with open source.
Re:Software Freedom Doesn't Kill People, Security Through Ob... by nickpsecurity on Sunday August 14, 2016 @05:54PM (#12287520)
"Good methods for what? Not certainly to certify that each combination of inputs gives a certain output, as you can read from that paper."

You're ignoring the biggest issue in your whole tangent: real-world engineers with sense won't use NN's in systems unless the problem itself is non-deterministic & other methods don't work. Personally, I push control theory, state machines, ladder logic, whatever for applications like this. You can deterministically analyze them. Whereas, problems with too much uncertainty or requiring approximate solutions will always have an error margin. You personally have been pushing this thread to focus as much as possible on the NN case while ignoring 99% of real-world, use cases representing how the OP topic will usually play out. You also act like the inherent error in the tooling is bad when the problem domain itself has inherent error. So, I decided to address your tangent.

The first solution, safety monitors, goes way back in safety-critical use. That's to have a simple solution checking the ranges or other behavior of the NN that's able to override faulty NN with simpler logic. This can be done for GPS-guided drones that usually move in straight lines or smooth curves. The NN part is reset, diagnosed, or deactivated while the fail-safe system takes plane on simpler path or out of the area.

Second part checks the nets themselves. Again, given a non-deterministic problem domain, the nets will be non-deterministic with the best one can do getting the error margin way down or within fail-safe bounds. This other paper I found summarizes a lot of V&V methods for NN's:

https://people.cs.umass.edu/~btaylor/publications/PSI000008....

Safety monitors. Training data or updates with constraints built in. Automated, test-case generation tweaking expected data. Rule extraction. System visualization. Simulation. All should contribute to safer NN's. They can't be perfectly correct, though, because vision, complex flying, thinking... none of this is 100% accurate no matter what method you use.

Re:Software Freedom Doesn't Kill People, Security Through Ob... by chris_wot on Sunday August 14, 2016 @05:06AM (#12284906)
Says who? You may not be able to (I sure can't) but that doesn't mean there aren't those who can and do understand this sort of thing. And there are those who will literally sit down for months and years till they do understand it. Or they might rip it out and replace it.
Re:Software Freedom Doesn't Kill People, Security Through Ob... by jobigoud on Sunday August 14, 2016 @05:21AM (#12284956)
They will understand how it works in principle, not why it took this or that specific decision and they might not be able to exactly predict what it will do in an unforeseen situation.

The decision itself might be taken using an amount of data completely overwhelming to human being analysis, processed at inhuman speed.

Re:Software Freedom Doesn't Kill People, Security Through Ob... by tigershark on Sunday August 14, 2016 @05:24AM (#12284964)
Exactly, so having an open source autopilot systems would not have changed absolutely nothing, and the guy on the tesla watching Harry Potter would have been dead the same.
Re:Software Freedom Doesn't Kill People, Security Through Ob... by tigershark on Sunday August 14, 2016 @05:16AM (#12284936)
Guys, are you really saying that you think that you can certify every response of a NN for every input? Especially for NN as complicated as autopilot systems? Are you aware that if that was true then you would be the richest men in the world given that everyone would come to you to write a perfect algorithm to drive a car?
Re:Software Freedom Doesn't Kill People, Security Through Ob... by ptero on Sunday August 14, 2016 @08:39AM (#12285359)
Certifiable reliability is not the only goal.

But even in safety critical tasks I have seen NNs and MDPs used to derive a good enough solution, then training turned off and the system certified (e.g. by running a large gold standard collection of test cases)

Re:Software Freedom Doesn't Kill People, Security Through Ob... by nickpsecurity on Sunday August 14, 2016 @07:12PM (#12287746)
This isn't true at all. What's necessary is code written in a verifiable way, qualified reviewer assessing it, trust in that reviewer/assessment, and way to ensure you're using what was reviewed in evaluated configuration. It's how main, security certifications work if you do it at high-assurance level. Lower levels are paper BS or simply inadequate given short-term, code review only catches so much.

I broke it down here with a quick revision just now:

http://pastebin.com/pjc0cueA