News for hackers. Stuff that matters.
Re:Leela Zero – Go engine with no human-provided knowledge by vog on Thursday November 16, 2017 @04:51AM (#15711425)
I applaud the work that went into this!

But at the same time I feel sad for the state of science where working code is still not considered part of the paper, so the code is almost never published, let alone published as Free Software for other scienists (and hobbyists) to tinker with it, to fix it, to improve it, and to adapt it.

For example, why isn't it possible for everyone interested to let Leela Zero play against AlphaGo Zero, on their own hardware (or their university's hardware)?

Re:Leela Zero – Go engine with no human-provided knowledge by dkersten on Thursday November 16, 2017 @05:45AM (#15711644)
> so the code is almost never published

Even worse is that a friend of mine who is a researcher in neuroscience/BCI has found that many of the papers couldn't be reproduced at all with flawed or incorrect conclusions. Since they typically don't publish their code, its very difficult for others to check the correctness.

Re:Leela Zero – Go engine with no human-provided knowledge by vog on Thursday November 16, 2017 @05:48AM (#15711664)
Moreover, in this context "code" is not just Python/C++/Java, but also spreadsheets!

Publishing spreadsheets is perhaps an even more urgent issue than publishing other types of code.

And of course the underlying data (anonymized) in case of statistical elicitation, but that's another topic. (Open Access, Open Data and Free Software are very closely related, though).

Re:Leela Zero – Go engine with no human-provided knowledge by fjfaase on Thursday November 16, 2017 @07:25AM (#15712121)
And even when it is made available, it is of very poor quality. I heard about someome who studied the programs used for analyzing DNA sequences. While tinkering with it, he found a bug, fixed it and submitted a pull request. Two years later, the pull request was still not merged by the maintainer of this 'standard' package. He also told that the software was of a poor quality. For example, in theory, four base pairs could be stored in one byte, instead they are simply storing the letters A,C,T, or G in a single byte for each pair.
Re:Leela Zero – Go engine with no human-provided knowledge by dkersten on Thursday November 16, 2017 @08:37AM (#15712492)
Yeah, that’s what my friend was saying too. He mentioned one often-cited paper which, when he fixed a bug he found in the code, no longer supported the papers conclusion at all. He also complained about the poor quality in a lot of the code. :-(
Re:Leela Zero – Go engine with no human-provided knowledge by znfii on Thursday November 16, 2017 @12:47PM (#15714746)
I haven't tested this myself, but from what I have heard it is not necessarily a good idea to store several different things in a single byte like this. While it may consume less memory there is a performance cost for each access since you may need to do masks and shifts.

Of course, there is a non-trivial performance trade-off with more cycles but more objects fits in cache, etc.

Some quick googling gave for example this source which seems to hint that under some situations it is not worth it at least https://stackoverflow.com/questions/4240974/when-is-it-worth...

Lastly, it seems like being quite harsh to call the code being of poor quality because of something like that. Unless it really is very important to get the extra memory, just the maintenance cost of packing things efficiently might make it not worth it. Even production code bases such as the JVM will happily use a single byte (or even more?) to store a boolean, and such things probably occur all over the place.

Re:Leela Zero – Go engine with no human-provided knowledge by kirillseva on Thursday November 16, 2017 @04:26PM (#15716779)
unless the author uses memory in the most efficient way possible (preferably writing LLVM-IR directly) it will be hard for reviewers to run DNA analysis algorithms on their smart fridges /s
Re:Leela Zero – Go engine with no human-provided knowledge by raverbashing on Thursday November 16, 2017 @06:24AM (#15711840)
Really, the academic world is sometimes obsessed with pseudocode.

It might make some didactic sense, but it makes fewer and fewer sense as times go by when you can ship the actual code and shows actual results

Re:Leela Zero – Go engine with no human-provided knowledge by deepnotderp on Thursday November 16, 2017 @01:38PM (#15715256)
Even worse is when you realize that's a growing concern in deep learning.
Re:Leela Zero – Go engine with no human-provided knowledge by pmontra on Thursday November 16, 2017 @05:27AM (#15711561)
Agreed, but the main goal of the average researcher is getting the paper published. The code is a byproduct and probably written not too well (they are pro of another profession) and maybe they don't know how to properly distribute it. Furthermore their publishing channel is made for paper / PDF, not source code.

In this case the author is DeepMind/Google, so they know how to code and how to distribute it.

However we need not only the code but also the network. Leela Zero is code but more importantly an effort to spread the effort of training the network among many people. Each one volunteers some hw (CPU/GPU) and contributes a part of the training runs. The README states that the estimated training duration on commodity hardware would be 1,700 years. We're not that patient :-)

Re:Leela Zero – Go engine with no human-provided knowledge by ForHackernews on Thursday November 16, 2017 @05:33AM (#15711588)
Could you design a cryptocurrency that used neural network training as part of a proof-of-work function? Then you'd actually get something useful at the end instead of brute-force hash collisions.
Re:Leela Zero – Go engine with no human-provided knowledge by zdkl on Thursday November 16, 2017 @05:39AM (#15711613)
You'd need to find some task that is expensive to compute, but trivial to verify to "port" current PoW methods, but the hard part is finding a task that fills this condition and is useful to actually compute.

In any case, you'd still be running a competition between nodes for fastest/best solution, with lots of duplicated effort and consumed resources. That is the key part of crypto consensus afaiu, and regardless of "backend task", there's always going to be waste in this architecture.

Re:Leela Zero – Go engine with no human-provided knowledge by yorwba on Thursday November 16, 2017 @05:53AM (#15711699)
The expensive task in machine learning is the training, it can be verified by checking for performance improvement on a validation set. You can make use of the duplicated effort if everyone trains on different examples, then their learned models can be merged/ensembled together for better performance.

That's the ideal situation when everyone plays by the rules. To make that scheme work as a "crypto"currency, you'll also have to make sure that breaking the rules doesn't bring a benefit. E.g. the validation examples can't be known beforehand, otherwise someone could just overfit to them to get a higher score with less work. But I think it might be possible to make it work.

Re:Leela Zero – Go engine with no human-provided knowledge by JustFinishedBSG on Thursday November 16, 2017 @11:30AM (#15713967)
> it can be verified by checking for performance improvement on a validation set.

Not all gradient updates improve performance

Re:Leela Zero – Go engine with no human-provided knowledge by yorwba on Thursday November 16, 2017 @10:20PM (#15718638)
Not all attempts to find a Bitcoin hash of the right difficulty succeed either. Miners would have to continue training until they are fairly certain that it has resulted in an improvement.
Re:Leela Zero – Go engine with no human-provided knowledge by bufferoverflow on Thursday November 16, 2017 @05:45AM (#15711643)
Which should be possible. For instance, your PoW could be "design a Go game bot neural network that's better than the system currently challenges you with".

Hard to make, easy to verify.

Re:Leela Zero – Go engine with no human-provided knowledge by vog on Thursday November 16, 2017 @05:57AM (#15711719)
> easy to verify

This may be a lot harder than you think.

How do you decide that one AI plays better than another? By playing lots of games. These games must be random, i.e. not known beforehand, otherwise the new AI could cheat.

But then, every cryptocurrency node would use a different set of game settings for validation. Which means that some (unlucky) nodes may reject the new AI while others accept it. The result are regular net splits, making it very hard to establish an keep the global consensus that is absolutely needed for a stable cryptocurrency system.

The core issue here is that validation must be 100% deterministic. Each node must run exactly the same calculations for validation, otherwise the consensus network is doomed.

Re:Leela Zero – Go engine with no human-provided knowledge by setr on Thursday November 16, 2017 @07:17PM (#15717938)
N seeds for the prng algorithm shipped with the program to be verified. Each seed provides for one game. N is sufficiently large to effectively remove "luck" for the skill comparison, and sufficiently large to make finding enough seeds allowing wins for a cheating weaker player more difficult than generating a better go player

Im assuming all games played against the incumbent; seeds are basically proof of wins. I guess you could still cheat though by copying the incumbent, get a 50-50 ratio of games, and then just find enough winners to make it 49-51.

Re:Leela Zero – Go engine with no human-provided knowledge by romaniv on Thursday November 16, 2017 @11:47AM (#15714146)
I think this is a very interesting idea. Except I would probably use a random forest instead of ANN for something like this. It is trivially parallelizable, you can train on subsets of features, you can slit and merge your ensembles/forests and so on.
Re:Leela Zero – Go engine with no human-provided knowledge by alexott on Thursday November 16, 2017 @05:41AM (#15711624)
it becoming better in the recent times with projects like http://www.gitxiv.com/