Truthfulness is on totally different philosophical footing than consciousness, so the analogy doesn't hold up. There is no "hard problem of epistemology", but there is a hard problem of consciousness. There exist criteria for evaluating truthfulness (that we may reasonably disagree on) that can be applied to LLM output.
I think the Gettier problems are at least epistemological problems with some weight, at least as long as you're in the Justified True Belief sort of camp (which I think many of us start out as).
If you haven't seen Gettier problems before, I'd start with the generalized problem and work out from there: https://en.wikipedia.org/wiki/Gettier_problem#The_generalize...
I think the whole point of the philosophical concept of "bullshit" is that it's orthogonal to the truth. It doesn't matter whether the words end up being correct or not correct, it's still bullshit because the speaker doesn't know or care if it's actually true.
Many philosophers argue that knowing whether something which appears to be conscious really is conscious is a meaningless question. If there is no way to distinguish consciousness from non-consciousness from the outside, then you may as well just say that the thing is conscious.
I think we may be approaching the same question with regard to truth. If it appears to be truth from the outside, is that the same thing as being truth? Is there any point in knowing wether it was produced from a thinking mind or not?