News for hackers. Stuff that matters.
Re:Missing data hinder replication of artificial intelligenc... by lettergram on Friday February 16, 2018 @10:11AM (#16392840)
Hell, I did an analysis on "Causality in Crypto Markets":

https://blog.projectpiglet.com/2018/01/causality-in-cryptoma...

I provided code and the data (since I knew no one would believe me, without evidence). Why do we believe researchers who have more incentive than I do to publish? Their jobs require publication, or at least are greatly improved by it, thus they have every incentive to be less than honest about their "experiments". Most of the "A.I. revolution" isn't about algorithms, it's about having more data and the right data. That's why Google and Facebook can out compete in research, as much as they do. They have all the data.

Personally, I've almost made it a life mission to try and replicate every study I can. After watching people in labs (private and public biology labs), just straight up make up data - I never believe anything.

I also wrote some stuff on backtesting (which in industry is viewed as the gold standard, but IMO is deeply prone to flaws): https://blog.projectpiglet.com/2018/01/perils-of-backtesting...

Having worked in industry and worked in academia... I don't believe 90% of the research I hear.

Re:Missing data hinder replication of artificial intelligenc... by madenine on Friday February 16, 2018 @11:33AM (#16393496)
Its gotten to the point where the first thing I do when reading a paper is skim for their training data.

If its not from a public/standard/replicable data source my bullshit meter hops up a couple steps.

"We improved on the current SotA by 15% but you can't see the data or the code - but trust us, its for real"

Re:Missing data hinder replication of artificial intelligenc... by MaxBarraclough on Friday February 16, 2018 @01:36PM (#16394626)
A similar line of thinking to the always provide the source code behind your paper position.
Re:Missing data hinder replication of artificial intelligenc... by dmreedy on Friday February 16, 2018 @02:06PM (#16394936)
In fact, I'd go so far to say that data is source code in the case of Machine Learning; it just goes through a kind of compilation step.
Re:Missing data hinder replication of artificial intelligenc... by MaxBarraclough on Friday February 16, 2018 @05:50PM (#16397260)
Careful now, don't tempt the LISPers.
Re:Missing data hinder replication of artificial intelligenc... by IntronExon on Friday February 16, 2018 @10:20AM (#16392906)
You might enjoy this site if you don’t already: http://retractionwatch.com

(No affiliation)

Re:Missing data hinder replication of artificial intelligenc... by TuringNYC on Friday February 16, 2018 @12:00PM (#16393719)
I'm honestly amazed at how some papers (esp in medicine) magically train deep nets using 1000 or 2000 cases and get absurdly high 90s AUC.
Re:Missing data hinder replication of artificial intelligenc... by madenine on Friday February 16, 2018 @01:47PM (#16394740)
I imagine getting samples/training sets that don't have some hefty source of bias is really tough in medicine, and probably leads to over-fitting.
Re:Missing data hinder replication of artificial intelligenc... by nonbel on Friday February 16, 2018 @10:15AM (#16392861)
Yes, I have come to much prefer a blog post and/or github to any "published" paper. It is almost like these journals are actively impeding progress at this point in every way possible.
Re:Missing data hinder replication of artificial intelligenc... by ashelmire on Friday February 16, 2018 @12:27PM (#16393957)
Ditto. Data can be manipulated, so can results. Small changes demanded by a PM can have a huge impact on results. The input data and the way it's handled are going to have a huge impact on any ML result.
Re:Missing data hinder replication of artificial intelligenc... by d0lph on Friday February 16, 2018 @11:31AM (#16393485)
There should be a major hit to the reputation of anyone publishing fake data. I am a bit surprised studies aren't independently reproduced immediately.
Re:Missing data hinder replication of artificial intelligenc... by lettergram on Friday February 16, 2018 @01:43PM (#16394702)
I considered starting a non-profit company around that, basically - you could donate to reproduce research. Then we could provide grants to reproduce the research. Kind of like a bounty system.

Even registered the domain: http://researchdonor.com/

I know a few people with terminal illness who would donate large sums of money for reproducible research.