News for hackers. Stuff that matters.
Re:Exam Credit for Knowing What You Do Not Know by whack on Saturday March 04, 2017 @03:12PM (#13792702)
There was a Decision Analysis class at Stanford that took this idea to an extreme. For each multiple choice option, you had to indicate the percentage confidence of that option being correct. The percentages for all 4 options had to add up to 100%. The score you got for each question = 1 + ln(confidence-assigned-for-correct-answer).

So if you were completely sure and gave the correct answer 100%, you would get a score of 1. If you gave it 90%, you would get a slightly lower score. If you "guessed wrong" and gave the correct answer 0% probability, ln0 is negative infinity, so you would instantly fail the entire assignment. It was a pain, but it really made you reassess how well you know stuff, rewarded people who truly knew their stuff, and burnt anyone whose confidence exceeded their actual abilities.

Re:Exam Credit for Knowing What You Do Not Know by Cyph0n on Saturday March 04, 2017 @04:21PM (#13792991)
Why would anyone ever put 0% then? I would just redefine "impossible" to be an arbitrary percentage that guarantees I don't nuke the question.

Quick question: is confidence taken in percent form (ln(90)) or decimal form (ln(0.9))?

Edit: I am aware of the implication of having percent values, guys/gals. But if it's in decimal form, a 5% confidence level will end up losing you the current question AND three others.

Re:Exam Credit for Knowing What You Do Not Know by cornstalks on Saturday March 04, 2017 @05:10PM (#13793168)
I'm not whack, but it must be in decimal form, or else putting 100% on a correct answer would yield ~5.6 instead of 1 (which whack says would be the score).
Re:Exam Credit for Knowing What You Do Not Know by derefr on Saturday March 04, 2017 @05:10PM (#13793167)
I assume that was part of the lesson: in Bayesian decision theory, an agent that assigns 0.0 or 1.0 confidence to something can no longer update on new evidence about that thing. Saying something is 0% likely means being so sure about that thing, that nothing—no matter how amazing—could or would ever change your mind.

Basically, no human is actually ever 0.0 or 1.0 sure of anything. Assign five or ten nines of confidence, but not 0.0 or 1.0. (It's a lot clearer when you remove the log scale and notice that a 1.0 confidence-value translates to you saying you're "infinitely confident", which seems ridiculous.)

Re:Exam Credit for Knowing What You Do Not Know by hyperpape on Saturday March 04, 2017 @06:50PM (#13793625)
I suppose this is a bad place to ask this, but given that you need to do math to update your credences and also that you can derive any proposition from Pr('2 + 2 = 4) != 1, how does this work?
Re:Exam Credit for Knowing What You Do Not Know by derefr on Saturday March 04, 2017 @08:04PM (#13793975)
Basically: probabilities and credences aren't the same thing. They're measured on the same scale, but probabilities can be used to reason about either facts or evidence, while credences are only "about" evidence (though potentially evidence about facts.)

To put that another way: it is "100% true" that 2 + 2 = 4 (in ZF set theory); just like it is "100% true" that the sun will rise today, because it already did (and otherwise the Monty Hall problem would have a different answer.)

But the credence in an agent's mind can and should only be informed by observations of these facts; and the "observing hardware" cannot be 1.0 trustworthy. (Your senses can be imprecise; the brain can be buggy or biased; the environment being observed can be the construction of a Cartesian "evil demon"; etc.)

Given an artificial agent with perfect knowledge (e.g. one who is able to directly entangle itself with the environment it is trying to predict, rather than just one able to entangle itself with evidence about the state of an environment), some things can theoretically be held to have 0.0 or 1.0 probabilities (e.g. events that already happened)—but given an embodied agent, the "maybe my processing is flawed" argument should still make these into non-infinite credences.

Re:Exam Credit for Knowing What You Do Not Know by hyperpape on Saturday March 04, 2017 @08:17PM (#13794028)
Thanks. I should've remembered that the connection to an agent's credences and their judgments of probabilities is complicated.
Re:Exam Credit for Knowing What You Do Not Know by platz on Saturday March 04, 2017 @09:42PM (#13794295)
If the Higgs field quantum tunnels to a lower energy level today in our vincinity, the sun will not rise tomorrow. Never put 100%
Re:Exam Credit for Knowing What You Do Not Know by derefr on Saturday March 04, 2017 @10:07PM (#13794365)
I didn't say "the sun will rise tomorrow"; I said "the sun will rise today"—as in, the (trivial) probability that the sunrise that already happened, will happen, looking forward from the more distant past to the more recent past. Given that we are talking about the probability of that event occurring from a reality where it occurred, the updated probability of it occurring is 100%. (But we cannot have 100% credence, simply because we can't actually be 100% sure we're in said reality.)
Re:Exam Credit for Knowing What You Do Not Know by jumpCastle on Saturday March 04, 2017 @06:40PM (#13793561)
But then what is my confidence that God exists? I would say zero because of the spaghetti monster argument
Re:Exam Credit for Knowing What You Do Not Know by Cyph0n on Saturday March 04, 2017 @07:08PM (#13793738)
He's right though, you can never be truly certain that some kind of God doesn't exist. But if you're referring to a specific God, then maybe.
Re:Exam Credit for Knowing What You Do Not Know by hexane360 on Saturday March 04, 2017 @05:00PM (#13793137)
If it was in percent form, a correct answer with 100% confidence would be 1 + ln(100) = 5.605 pts. Only ln(1) = 0.
Re:Exam Credit for Knowing What You Do Not Know by catern on Saturday March 04, 2017 @06:00PM (#13793375)
Yes, there's a class on the same subject at CMU with the same policy. Here's the document they handed out to explain the policy: http://cmu.io/~sbaugh/midterm_grading_function.pdf
Re:Exam Credit for Knowing What You Do Not Know by nickkoul on Saturday March 04, 2017 @04:20PM (#13792989)
In reality even in your most confident answers couldn't you put 97% on the one you believe is correct and 1% on all others ensuring that you will not receive an ln(0).
Re:Exam Credit for Knowing What You Do Not Know by Cyph0n on Saturday March 04, 2017 @04:22PM (#13792997)
It actually depends on how they take the confidence value. If it's percentage, I agree with you. But if it's in numerical form - i.e., 0.01 - you'd still lose that question and a few others.
Re:Exam Credit for Knowing What You Do Not Know by jacoblambda on Saturday March 04, 2017 @04:03PM (#13792914)
I have heard of this before and while it horrifies me as a student, I would love to take a class with a policy like this.
Re:Exam Credit for Knowing What You Do Not Know by hectormalot on Saturday March 04, 2017 @05:33PM (#13793271)
I had a professor that used a slightly different approach. A correct answer would yield you 1pt, but any of the 3 incorrect answers would subtract 0.5pts. Expected average result for guessing is negative, so he incentivized people to leave questions open if they were unsure.
Re:Exam Credit for Knowing What You Do Not Know by CaseyJParker on Saturday March 04, 2017 @05:37PM (#13793285)
The expected average for a straight guess with absolutely zero exposure to the material would still be slightly positive. If nothing else, there's intuition and ingenuity, which place a person just a touch above a truly random guess-generating-machine.
Re:Exam Credit for Knowing What You Do Not Know by hyperpape on Saturday March 04, 2017 @06:47PM (#13793597)
With four answers, you get .25 + (.75)(-.5) = -.125 for guessing.

Edit: yeah, I badly misread this.

Re:Exam Credit for Knowing What You Do Not Know by cercatrova on Saturday March 04, 2017 @07:17PM (#13793792)
That's what the parent is saying to be not applicable to the situation; humans are not simply true random guessing machines but are able to reason and eliminate choices, which would increase the expected value to be positive.