It has been awhile, but MSR used to be the place where people went and still published, while Google Labs was a black hole. I remember some Google guy giving a talk at NIPS, but it was just a presentation from a shared corporate slide deck, which did not fit in with the rest of the conference. Google attracted some really good machine learning folks (temporarily) by offering them boatloads of money and manpower to implement their ideas, not freedom to publish. Has that changed?

I don't know what you're talking about; Google researchers publish all the time and give presentations at machine learning conferences with plenty of interesting and unique detail. Google just didn't do much with machine learning in the old days- the pubs were mostly system infrastructure and data analysis.

Here's an oldie but a goodie: http://people.mbi.ohio-state.edu/datta.53/philtalk.pdf note the talk refers readers to the patent. At the time, reading Google's patents was the best strategy to learn what it was doing.

Yup, Google now has two teams publishing the very best ML papers alongside FAIR, Toronto, Montreal and some other places. The teams or Deep Mind and Google Brain.

To my knowledge, Apple ran a program called "The Apple University Consortium," among others, where they would tap into work of higher education research.

It's my understanding these efforts continue, you just don't hear a whole lot (if anything) about it publicly.

Check it out, you'll see Apple quietly listed on several university websites, CMU for example: http://www.cmu.edu/corporate/partnerships/cic/tenants.html

Some scientists just want to build cool things and not have to worry about publishing. Some of my colleagues over the years have moved to Apple and startups for this reason.

Perhaps so. Other scientists want to push the border of human knowledge.

There is a difference between pushing the borders of human knowledge and disseminating said knowledge, though.

I guess that depends on your interpretation of "human knowledge" then :)

By being a human, you would be able to expand human knowledge without letting other humans know about it. So yeah, it's a matter of definition;P

Exactly. The first, Without the second, is not called science.

If the scientists in question are applying the scientific method as a means to "build cool things", can they not be said to be doing science?

One of the key tenets of the scientific method is reproducibility. People can't reproduce your results if they don't know what your results are.

Reproducibility does not necessarily mean that other people should be able to reproduce the results in question (which would depend upon them knowing the relevant information in the first place) - but that an experiment can be replicated.

I would agree that publishing results is better for several reasons (ethics, pragmatism, probably more thorough evaluation, etc.), but the definition doesn't really deal with the public at all.

TLDR; Reproducibility != public reproducibility.

It doesn't have to be "arbitrarily many people". Just people other than the original scientist. So colleagues at Apple still qualify.

There are tons of secret research behind any company (and governments), and it's still "science".

(And conversely, lots of published scientific papers are actually not reproducible, but for most of them nobody bothers -- even if other scientists cite them as accurate).

It's just the conclusions that need to be reproducible. In fact, science as a whole works better when researchers devise alternate experiments to test the same conclusion.

For example, say I measure gravitational acceleration on Earth by dropping a feather repeatedly. I get a number and I publish it. If other scientists just repeat my exact experiment, they will get a similar number and my result will be confirmed. Yay science!

But if scientists decide to check my number by dropping a wide variety of other objects, they will illustrate the flaws in my original experiment and everyone will get a clearer picture of gravity.

So it's not about just reproducing an experiment; scientists seek to test and expand upon each others' results.

People studying machine learning are not like people running a psychology experiment. If you give the same code, data and computers to two different teams, they will get the same results, and they will understand why that happens.

That's because a machine learning system has been built up from known principles and hardware; whereas the human brain is not fully understood, so researchers must work their way down from observed evidence to induce theories.

Luckily bugs don't exist in machine-learning software!

What is a bug, anyway? It means the system is not working properly, which implies that folks know in advance how the system is intended to work.

When physicists were struggling to explain radiation, they didn't call it a bug in the atom. It was their understanding, not the system they studied, that needed fixing.