Side note: I often say that you know when the physics undergrads start quantum mechanics because they all become alcoholics. The joke here (which has a lot of truth to it btw) is that your intuitive framework breaks down and stops helping you, so you suck it up, drink, and just trust the math. Truthfully, for any study, once you hit a sufficient level, this kinda ends up being true. You get introduced to concepts at a simple and abstracted level, which seems sufficient for modeling the world (often this is what you learn in undergrad domains, not just physics). But once you get to enough expertise you find that what seemed simple before simply gets dominated by the nuances and complexities of the system. Truth is that if anyone is trying to convince you that something is simple (especially yourself), then they are trying to deceive you -- intentionally or unintentionally (often fooled themselves). There's a reason you'll find huge books on how to pick the best screw, o-ring, or whatever seemingly simple thing.
I guess it wouldn't be a clique if it wasn't well known but people fail to internalize it: The devil is in the details.
Not climate scientist, but researcher in ML (another highly hyped and arguably more noisy field).
After Bourbon, the hard truth is expertise + don't.
Unfortunately journals/conferences/venues are an extremely noisy mechanism which in general aren't realistically much better than arxiv (at least over here). Peer review isn't journals/conferences, it is peers reading and evaluating the works. Reviewers are doing a service and often not giving a work significant time as they got other stuff to do. The only real way to know if a paper is valid or not is to be an expert in the subject matter (more focused than the field) and to read it __and the code__. All too often it is a nuanced point that is the crux of a paper, and would be entirely missed if you're not deep in that subcommunity. I know this answer sucks, but that's what it is.
Fwiw, I wouldn't realistically change your strategy as a layman. It's probably the best you got. Maybe only thing is don't think "journal publication == 100% true" but rather "journal publication == probably right." The process is noisy and there's no way around this.