Quick question: is confidence taken in percent form (ln(90)) or decimal form (ln(0.9))?
Edit: I am aware of the implication of having percent values, guys/gals. But if it's in decimal form, a 5% confidence level will end up losing you the current question AND three others.
Basically, no human is actually ever 0.0 or 1.0 sure of anything. Assign five or ten nines of confidence, but not 0.0 or 1.0. (It's a lot clearer when you remove the log scale and notice that a 1.0 confidence-value translates to you saying you're "infinitely confident", which seems ridiculous.)
To put that another way: it is "100% true" that 2 + 2 = 4 (in ZF set theory); just like it is "100% true" that the sun will rise today, because it already did (and otherwise the Monty Hall problem would have a different answer.)
But the credence in an agent's mind can and should only be informed by observations of these facts; and the "observing hardware" cannot be 1.0 trustworthy. (Your senses can be imprecise; the brain can be buggy or biased; the environment being observed can be the construction of a Cartesian "evil demon"; etc.)
Given an artificial agent with perfect knowledge (e.g. one who is able to directly entangle itself with the environment it is trying to predict, rather than just one able to entangle itself with evidence about the state of an environment), some things can theoretically be held to have 0.0 or 1.0 probabilities (e.g. events that already happened)—but given an embodied agent, the "maybe my processing is flawed" argument should still make these into non-infinite credences.
Edit: yeah, I badly misread this.
So if you were completely sure and gave the correct answer 100%, you would get a score of 1. If you gave it 90%, you would get a slightly lower score. If you "guessed wrong" and gave the correct answer 0% probability, ln0 is negative infinity, so you would instantly fail the entire assignment. It was a pain, but it really made you reassess how well you know stuff, rewarded people who truly knew their stuff, and burnt anyone whose confidence exceeded their actual abilities.