>How can you get transparency from an LLM? They're complete black boxes by architecture.

You just ask them. I literally just prompted ChatGPT (3.5) to think about its answer again and explain what's wrong with it, and it did. Kept going 2 or 3 times, until it was sure it wasn't doing anything wrong, and magically the script it produced at that stage just worked.

But can you believe that these are the correct explanations? Perhaps it just confabulated an explanation that sounds likely given your prompt, and is not the actual reason it told you what it did?

That'll just give you made up rationalizations, like asking a human to explain an emotional subconscious choice.

For inference? Not really. LLM observability startups are a thing. Arize.ai probably the most well known.