You are talking about this anecdote https://en.m.wikipedia.org/wiki/Gell-Mann_amnesia_effect

And I partially agree! But do not throw out the baby with the bathwater! I am a physicist myself and have seen how the Gell-Mann effect has become less problematic over time, especially in respected journals that want to be taken seriously by scientists: Just make a decisive effort to 1) go out of your bubble and 2) learn about logical fallacies.