Jev Does Not Play Dice: 83% probability, 19% accuracy on a hidden fair die roll

Re:Jev Does Not Play Dice: 83% probability, 19% accuracy on ... by kantahayashi on Thursday September 24, 2026 @09:37AM (#49830409)

Author here. I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%.

Code and data are here: https://github.com/KantaHayashiAI/jev-does-not-play-dice

Happy to answer questions!

[ Parent | on HN ]

Re:Jev Does Not Play Dice: 83% probability, 19% accuracy on ... by magicalhippo on Thursday September 24, 2026 @10:33AM (#49831194)

Maybe it watched Rosencrantz & Guildenstern Are Dead[1] once too many.

[1]: https://www.youtube.com/watch?v=C_TfdNAXOwE

[ Parent | on HN ]

Re:Jev Does Not Play Dice: 83% probability, 19% accuracy on ... by kantahayashi on Thursday September 24, 2026 @10:55AM (#49831512)

I found that in the play it's 92 heads in a row [1], and in my coin test Jev returned around 92%. Maybe Jev was actually trained on it.

[1]: https://en.wikipedia.org/wiki/Rosencrantz_and_Guildenstern_A...

[ Parent | on HN ]

Re:Jev Does Not Play Dice: 83% probability, 19% accuracy on ... by __m on Thursday September 24, 2026 @09:42AM (#49830458)

why would you use ai for that though? how does that translate to other tasks?

[ Parent | on HN ]

Re:Jev Does Not Play Dice: 83% probability, 19% accuracy on ... by kantahayashi on Thursday September 24, 2026 @09:56AM (#49830671)

I used a die as an example of problems whose answers can't be known at all, unlike problems with correct answers like MMLU questions. In a more practical test, a document said "30% risk" and Jev returned 5%. The point of the post is that Jev is weak at some kinds of tasks, so you should check the calibration for your own use case.

[ Parent | on HN ]