On indeed. Heisenberg used to say, "data does not imply theory."[1] Once you deeply internalize that, you realize you can't really be that, ahem, certain about anything in scientific thought outside of the limited domain of how scientific truth is defined (namely that the model has predictive value, which is entirely different from whether it has ontological value).
P values are estimates of the probability that a given observation is likely to have come about by chance, rather than due to a hypothetical model. They have many problems and it is best to think about them with a solid understanding of probability.
If there was a case where you have as many conflicting results as you say, then it would be indicative of a very active scientific community and problem. People would be working hard to resolve the problem!
What does it say about X when you have 10 studies saying X is true with p < .05, and another 10 studies saying that X is false with p < .05? It seems like the closer you come to actually taking p values at face value given conflicting results, the amount of things that can be considered to be 'scientifically proven' rapidly approaches zero.
Also, while I do like norvig's essay, all the errors he goes through are fairly pedestrian. If you read the modern scientific literature on research, the rabbit hole goes vastly deeper. (Assuming researchers researching researchers are more trustworthy than researchers.)