> The point being, unless it's possible to conclusively and infallibly prove you are not hallucinating, you can't lay claim to having knowledge an AI is hallucinating, or not.

This is very much true for more irrational (or less deterministic) "thoughts" in certain languages. However, if I asked you to write JavaScript or Python to "prove" your thoughts on the answer, it would be a simple matter of running the code you output on a state machine to determine the truths of the query.

  super-frog-of-adventure~> !calc 1+1
  athletic-giraffe|> 2
  super-frog-of-adventure~> !calc one plus one and output the answer
  athletic-giraffe|> Writing code.
  // filename: synth_calc_plus.js
  // version: 0.1.1
  // description: simple addition calculation
  var num1 = 1;
  var num2 = 1;
  var answer = num1 + num2;
  Sidekick(answer + " (units)", ", ");
  athletic-giraffe|> 2 (units)
  super-frog-of-adventure~>