If we do somehow try to apply your analogy, it would indicate that the LLM output is flawed in a way we cannot scrutinize -- the hidden failures that we aren't detecting (why? It's not specified, I am assuming because we didn't check to see if the tool was "broken" and not meeting some unspecified quality-level; that is, it's an unknown unknown failure mode).
This doesn't really comport with the LLM scenario, where the output is fully viewed, and the outputs are widely understood (that is it is a known failure mode).
This is more closely related to a computing service -- of which you are an active user of. You are using a "broken" computer right now, according to your definition of broken correct ?
Yes, it does. People are regularly using LLMs to brief themselves on topics they are ignorant about. Are you actually serious?
>This is more closely related to a computing service -- of which you are an active user of. You are using a "broken" computer right now, according to your definition of broken correct ?
Doesn't really matter, because I'm not relying upon the computer for the verity of its functions
Your CPU, right now, has known defects. It will produce the wrong outputs for some inputs. It seems to meet your definition of broken.
Do you agree with that premise ?