The lack of legitimacy is not an imagined made-up fear of lay non-expert people. It has been in the news that over half of psych papers are not reproducible. Maybe thats why people have a hard time accepting related fields as science. Sure, its always better to counter flawed science with better science. But if half of your accepted papers are not reproducible then it does cast a doubt on whether its a science to begin with.