Ideally this is true. At the worst though, you'll end up with a peer reviewer that gets pedantic or one who's a member of a citation cartel [0] that demands that you cite his/her friends papers unnecessarily as a quid-pro-quo leading to publication. It's difficult to say how common this is in practice since academia is generally terrible about self-reflection, but I observed concerning evidence of this when I was still in science. In some fields at least, there's ample evidence [1] that peer review provides inadequate QC controls for methods sections, making it literally impossible to replicate results due to entire processing steps being omitted. This is why post-publication services such as PubPeer have become more and more relevant, since it has become obvious as time has elapsed how woefully inadequate the initial screening process is.
[0] https://academia.stackexchange.com/questions/81261/reviewer-...
So while this definitely does happen (and to different degrees in different fields) there are some checks and balances, too.
I could be naive, but my impression was that there are almost too many degrees of freedom in the peer review process that could allow for this kind of chicanery, and that it would almost be preferable to have criteria for acceptance laid out in advance (i.e., exactly what steps, metrics, and citable articles were necessary) and use those the way an actuary would. In some ways I almost think that practices like preregistration serve as a better first-pass indicator of whether or not research is publishable.
I personally would like more use of pre-registration (there is something about that on the front-page right now. But also I think deanonymzing peer review and publishing the reviews along with the papers could be very interesting. Obvious drawbacks as well of course but I think it has potentially massive upsides.
It provides value, indeed most of my papers have been improved by going through peer review. But it's good to be conscious of what it can and can not do.
One thing that it does do is signal intent. It signals that you are willing to engage with the expert community, and specifically their feedback and criticism.
Finally, there is of course a fallacy in saying "not trustworthy either". If 99% of random blog posts are wrong, but 60% of published papers are wrong, then a random paper is 40 times more likely to be correct than a random blog post and it's perfectly sensible to be hesitant about citing the latter.
Finally, scientists have always cited outside of traditional sources and continue to do so [1]. I see no evidence for your assertion that "appeal to authority" has become more prevalent or important than it used to be. With all the ongoing open science initiatives it's rather the opposite.
[1] Personal communication with various scientists, 2018