spekcular

Is the "optimal Julia workflow" written down anywhere?

You may be interested to know that this exact objection has been made in the philosophical literature. See "Causal Fundamentalism in Physics" by Zinkernagel (2010). Available here: https://philsci-archive.pitt.edu/4690/1/CausalFundam.pdf

At the end, the author notes (as you do) that if you consider a finite difference equation with small time steps, there are no pathological solutions. He also mentions that Newton takes this difference equation approach when solving problems in his Principia.

See also "The Norton Dome and the Nineteenth Century Foundations of Determinism" by van Strien:

>> Abstract. The recent discovery of an indeterministic system in classical mechanics, the Norton dome, has shown that answering the question whether classical mechanics is deterministic can be a complicated matter. In this paper I show that indeterministic systems similar to the Norton dome were already known in the nineteenth century: I discuss four nineteenth century authors who wrote about such systems, namely Poisson, Duhamel, Boussinesq and Bertrand. However, I argue that their discussion of such systems was very different from the contemporary discussion about the Norton dome, because physicists in the nineteenth century conceived of determinism in essentially different ways: whereas in the contemporary literature on determinism in classical physics, determinism is usually taken to be a property of the equations of physics, in the nineteenth century determinism was primarily taken to be a presupposition of theories in physics, and as such it was not necessarily affected by the possible existence of systems such as the Norton dome.

The author is a professor at Ohio State University, specializing in condensed matter theory. His CV list two works on superconductors and a dissertation on supercapacitors.

There's also a C-R bound for the unbiased case.

But here the issue is the uncorrected multiple testing and under-reporting of results, not the p-values themselves. Any criterion for judging the presence of an effect is going to suffer from the same issue, if researchers don't pre-register and report all of their analyses (since otherwise you have censored data, "researcher degrees of freedom," and so on). This is really a a problem with the design and reporting of studies, not the analysis method.

AIC is an estimate of prediction error. I would caution against using it for selecting a model for the purpose of inference of e.g. population parameters from some dataset (without producing some additional justification that this is a sensible thing to do). Also, uncertainty quantification after data-dependent model selection can be tricky.

Best practice (as I understand it) is to fix the model ahead of time, before seeing the data, if possible (as in a randomized controlled trial of a new medicine, etc.).

How else do you propose to construct procedures that control type I error and evaluate their properties?

I don't understand. CIs are equivalent to computing a bunch of p-values, by test-interval duality. Should I interpret your points as critiques of simple analyses that only test a single point null of no effect (and go no further)? (I would agree that is bad.)

The items you listed are certainly problems, but p-values don't have much to do with them, as far as I can see. Poor power is an experimental design problem, not a problem with the analysis technique. Not reporting all analyses is a data censoring problem (this is what I understand "specification curve analysis" to mean, based on some Googling - let me know if I misinterpreted). Again, this can't really be fixed at the analysis stage (at least without strong assumptions on the form of the censoring). The replication crisis is a combination of these these two things, and other design issues.

I think if hypothesis testing is understood properly, these objections don't have much teeth.

1. Typically we use p-values to construct confidence intervals, answering the concern about quantifying the effect size. (That is, the confidence interval is the collection of all values not rejected by the hypothesis test.)

2. P-values control type I error. Well-powered designs control type I and type II error. Good control of these errors is a kind of minimal requirement for a statistical procedure. Your example shows that we should perhaps consider more than just these aspects, but we should certainly be suspicious of any procedure that doesn't have good type I and II error control.

3. This is a problem with any kind of statistical modeling, and is not specific to p-values. All statistical techniques make assumptions that generally render them invalid when violated.

Is there a modern reference you would recommend?

My argument is that key claims of the paper are factually incorrect. For example, there was no "confession" by Karplus and Kroll. (At least, none is cited as far as I see - please let me know if I am missing something.)

If a mathematical technique helps understand and predict nature, why not use it? The quote criticizing the method misunderstands it. It's certainly not "wrong" or "absurd."

I dislike the paper because it is sensationalist and highly misleading. For example, as far as I can tell, Karplus and Kroll didn't "confess" to anything (no direct quote is provided in the paper); we just have a secondhand assertion by Feynman. Nothing they did is "fraud," as the author claims - this is false and defamatory. Further, the issue got wrapped up by Petermann in 1957; the author is just annoyed that he didn't publish full details of the calculations. The suggestion that there is somehow lingering uncertainty is just wrong.

I no particular wishes for the author, other than that he cease writing bad papers.

This has already been posted a few times: https://news.ycombinator.com/item?id=31431033

The summary is that the author doesn't know what he's talking about.

I see - I understood "liberal arts program" above to mean a liberal arts college in general (typically offering a mathematics major). I agree that this reading list is better suited for something like "history of math for humanities students."

Yes, it is substantially different with respect to rigor and content than standard undergraduate mathematics programs. It covers a few historically important texts and does not teach (if those texts are any indication) most of what is usually taught in an undergraduate math degree. (A poster above writes: "Freshman math was almost entirely the study of Euclid and Nicomachus.")

I believe the person you're replying to understands that. The claim is that this approach is really bad way to learn math.

As someone who has learned and taught a lot of math, I agree with that claim.