Here's the idea. If you set alpha = 0.05, you will declare statistically significant any result that gets a p value of 0.05 or less. When there is no statistically significant difference to be found, you will have a 5% chance of falsely detecting one.
But crucially, this applies on average to all tests you conduct with this alpha level. Even if an individual test gets p = 0.000001 or p = 0.04, the overall false positive rate will be 5%.
More succinctly, it doesn't make sense to ask for the false positive rate of a single test. What does that even mean? You can only ask for the false positive rate of a procedure you use many times. So you can't get p = 0.01 and declare this means you have a false positive rate of 1%.
2. As far as what the difference is, I have to admit I have not found a memorable phrase to explain it. Both Goodman in "Toward Evidence-Based Medical Statistics" and this paper give a similar wording of the difference, which i would rephrase as follows.
The p-value works by inference, taking the data and assigning a probability to that data, not to an hypothesis. It is to be used to corroborate our disbelief in the null, i.e. informally. To use inference to test hypothesis one must use Bayes' Theorem, i.e. Bayes factors, and introduce prior probabilities. Hypothesis testing à la Neyman and Pearson is deductive process, in which one does assign probabilities to the hypotheses, paying the price of only being able to minimize the errors commited, not to draw inferences.
One should also confront "In Fisher’s approach the researcher sets up a null hypothesis that a sample comes from a hypothetical infinite population with a known sampling distribution." with "Neyman–Pearson results are predicated on the assumption of repeated random sampling from a defined population."
Given the natural predisposition by students to works in an inferential manner, it may be wise to bite the bullet and teach Bayes factor instead of the p-value cargo cult (this being a pedagogical choice, not an assessment of frequentism vs. bayesianism or a critique of p-values as conceived by Fischer).
1. At least at the beginning, it focuses excessively on the historical aspects of statistics. For example, it says that "most applied researchers are unmindful of the historical development of methods of statistical inference, and of the conflation of Fisherian a nd Neyman–Pearson ideas." To me, statisticians shouldn't /have/ to understand the history at all. For example, as a physicist, there is absolutely no need for me to understand the evolution of Ampère' theories, Faraday's theories, Maxwell theories, etc. to apply the laws of electricity and magnetism correctly.
2. The difference between p and alpha is central to the paper, but it doesn't seem to have a cogent explanation of what that difference is. (It's very clear who advocated for one and who advocated for the other, but that's not why the difference is important.)