It seems reasonable to me to expect journalists to have a better grasp of certain things than others (say politics or law instead of physics), and for this to vary across writers (one writer might have a background in chemistry; another in population biology).
I also think there's a certain variability of opinion on certain topics when you get to a certain level, and experts aren't always aware of this. That is, some (not all) tend to believe they are infallible and everyone else who disagrees is incorrect, or sometimes, "unscientific." There's probably several ways this could play a role, but I do think it plays some role in this.
I think this Gell-Mann amnesia is sort of a real bias that's interesting to think about, but I also think in reality it might not be quite what's being assumed in this essay.
Yes and no. It's more a method to rule out, not rule in. If a presenter gives bad information about something I know I can't trust them. If a presenter gives good information about something I know, it builds evidence that I can trust them. These two are different. The latter isn't complete trust and still maintains skepticism. At the end of the day, we have to have some trust. But that also doesn't mean absolute.
On subjects I'm not an expert in? Well I can't know.
The problem here is that I'm not an expert in everything. In fact, I'm not an expert in most things. I feel like people are taking this test as a perfect test. Far from it. There is no such test and never will be. But to everyone that is trying to counter this I'll ask you the inverse question: "Why should I trust someone when they've proven to mischaracterize a topic?" That's the heart of this metric.
If you've proven that you won't do enough research, I have reason to not believe you. If you've proven that you will do enough research, I have reason to believe you. It is statistical. Everything is. It is imperfect. Everything is. If you got suggestions about how to extend this and improve upon this, I'm listening. But I'm not just going to abandon it and start listening to people that have given me strong evidence to distrust them. That's a bit insane. It is also exactly what the Gell-Mann amnesia effect is making fun of: trusting people in things you don't know about that proved to be untrustworthy in things you do.
I've found this to be extremely important because many of the tough (and popular) topics we discuss today are extremely complex. But if people are only going into the basics and don't talk to experts about the complexity and nuance they will frequently make extremely bad conclusions. Science communicators are supposed to be taking the complex and nuanced topics that experts understand and distill it into something more palpable for the common (or even a bit nerdy) audience. But I do find that a lot of our extremely popular YouTube pop-sci channels or information based comedy shows (and even news) often don't actually seek out discussions with experts for clarity, even if they appear to or have the budget for it. It has really limited my source list of these channels and shows, but I also think it has really helped.