News for hackers. Stuff that matters.
Re:Mathematica v12.1 by vtail on Thursday March 19, 2020 @04:02PM (#22631082)
Many people on this and similar threads rightly question whether results received with the help of a closed source software are reproducible. I think it's a fair question, but these discussions often miss a couple of important points.

First, there is a question of theoretical vs. practical reproducibility: true, in _theory_, an open-source codebase with many millions of lines of code of highly complex transformation can be checked for accuracy by anyone. In _practice_, only very few insiders will have both technical understanding and time to verify the correctness of this or that algorithm, and everyone else would need to rely on their expertise. That situation is no different from using a proprietary system if we trust their authors.

Second, while it might be shocking for someone, most of the science, outside of maybe math, it's not _practically_ reproducible. Most of the articles are behind paywalls, don't have enough data to replicate their results (even economists much more often than not won't include their raw data or algorithms or both). Also, no one who is trying to build a scientific career would try to replicate someone else's results, especially if it requires some costly equipment, reagents, etc. - the rewards in the scientific community are for novel results. Replication crisis[1] describes this pretty well.

Most of the software that runs LHC at CERN is probably not inspectable by a regular Joe the physicist, yet somehow the physicist community somehow trusts their results, etc.

In summary, while an open-source software of Mathematica quality would be awesome in theory, I highly doubt it would be practical for a long time.

[1] https://en.wikipedia.org/wiki/Replication_crisis

Re:Mathematica v12.1 by reikonomusha on Thursday March 19, 2020 @04:09PM (#22631143)
Reproducible has a much simpler meaning than that you describe: it’s feasible for any scientist to recreate the result perpetually. While inspection of the methods might help one understand the result—and I think itself holds a great deal of value to especially mathematicians—just the simple ability to compute the answers yourself without binding yourself to a contract or paying $1000’s of dollars in license fees is a valuable aspect to science. Especially so when the calculations are at the level of pure algebraic manipulation.

Your (and Wolfram’s!) argument about “not needing to see the insides because really only 10 people in the world understand it anyway” should really be an argument for opening it up. If 10 people could understand it, and we must intersect that group with folks who can access the source, we are left in quite a dismal state of affairs.

Re:Mathematica v12.1 by vtail on Thursday March 19, 2020 @04:18PM (#22631255)
While I don't disagree with your definition of reproducibility, I want to point out that very few papers would satisfy that criteria, regardless of their use of a closed- vs open-source software.

Here is a recent example: Imperial College COVID-19 response team published an article[1] where they modeled different effects of non-pharmaceutical interventions, such as suppression and mitigation on the number of infected, deaths, etc. This is a very interesting result, but it's impossible to replicate their results in practice without contacting the authors, as their methodology is not enough to reproduce it.

While someone else[2] posted their own model that is very well documented and fully reproducible by anyone with a $230/year personal license.

While theoretically [1] is high science and [2] is not, in my own opinion [2] is better than [1]. I would love more science to be done and discussed that way. Ideally, using open-source software, but in practice, using Wolfram Language, in that case, is already good enough in my opinion.

PS. I'm not affiliated with Wolfram Research in any way.

[1] https://www.imperial.ac.uk/media/imperial-college/medicine/s...

[2] https://community.wolfram.com/groups/-/m/t/1901002

Re:Mathematica v12.1 by lonelappde on Sunday March 22, 2020 @10:17AM (#22655072)
Making an exact copy of an experiment is the lwest level of reproducibility. In science , reproducibility means reproducing the results with different components (people, tools, methodology), showing robust Independence form potential confounders.
Re:Mathematica v12.1 by vtail on Thursday March 19, 2020 @04:26PM (#22631354)
Regarding your second point - I don't have a horse in that race, other than tremendously enjoying my personal license that I pay for out of my pocket; but I found their arguments[1] quite reasonable. I think there was an HN discussion about this some time ago.

[1] https://blog.wolfram.com/2019/04/02/why-wolfram-tech-isnt-op...

Re:Mathematica v12.1 by reikonomusha on Thursday March 19, 2020 @04:34PM (#22631441)
I find the arguments confused and lacking. Many of those arguments conflate “open source” with “community driven”. Rarely did they make a compelling case as to how availability of the source code would hurt the customers.

And many, many amazing open source teams have demonstrated that even including community driven design in some cases does not hamper the quality of the product. Look at Apple and Swift, or Rust, as examples.

I read their arguments as “we do not know how to engage with a community effort because it’s not in our company DNA” under the facade of supposedly legitimate reasons.

Re:Mathematica v12.1 by vtail on Thursday March 19, 2020 @04:38PM (#22631481)
Swift is a great example, but don't forget that is essentially funded by Apple. It would look very different if it would be a true community effort.

Rust I'm not very familiar with: how much of their funding is dependent on Mozilla, at least for the core developer(s)?

Also, a language + a standard library has a (much!) smaller surface comparing to Wolfram's vision.

Re:Mathematica v12.1 by steveklabnik on Thursday March 19, 2020 @06:27PM (#22632417)
Mozilla funds a few things. Some stuff is funded by other companies. Mozilla pays the most folks to work full time, but the amount is tiny compared to the overall set of teams, let alone contributors.
Re:Mathematica v12.1 by vtail on Friday March 20, 2020 @12:34AM (#22634917)
Thanks Steve for the answer! HN is an amazing place where a random comment asking a question about Rust can get an answer from a core developer. I hope we will always keep it that way.
Re:Mathematica v12.1 by reikonomusha on Thursday March 19, 2020 @04:52PM (#22631608)
They have funding; that’s great! Exactly what I’m saying is that these projects manage to be community run (to some extent) while still remaining successful.

If Wolfram’s reason to not publish the source code is that they believe they would no longer have a business model, they should list that as the One Reason, not 12 gaslighting reasons.

Re:Mathematica v12.1 by vtail on Thursday March 19, 2020 @05:15PM (#22631816)
Well, they clearly think their model works for them. If you think otherwise, it's on you to convince them that they should change their model, or start your own effort if you think it will benefit humanity. I don't think they necessarily need to "defend" their business choices.
Re:Mathematica v12.1 by reikonomusha on Thursday March 19, 2020 @05:25PM (#22631913)
You're right, they don't need to defend it. But they shouldn't put out reasons that don't make sense. They ought to just put nothing out at all and operate as normal.
Re:Mathematica v12.1 by lonelappde on Sunday March 22, 2020 @10:19AM (#22655086)
I don't care what works for them. I care what works for society.