News for hackers. Stuff that matters.
Re:Jev Meets the Trolley Problem by nico on Thursday October 15, 2026 @02:28PM (#50108067)
Very cool, thank you for putting it together and sharing the data

Would you mind adding Jeffy to your benchmark?

https://github.com/nicobrenner/jeffy

https://jeffyclassify.com/blog/when-llms-fail-at-classificat...

Re:Jev Meets the Trolley Problem by evanjrowley on Thursday October 15, 2026 @02:48PM (#50108394)
Re:Jev Meets the Trolley Problem by nico on Thursday October 15, 2026 @03:28PM (#50108977)
Hasn’t seen it, authors not related

Looks very cool though. I’d definitely like to somehow “generalize” Jeffy to be more of a universal classifier. Although adhoc models pretty much always beat general ones at classification

Re:Jev Meets the Trolley Problem by zlandes86 on Thursday October 15, 2026 @02:32PM (#50108141)
running it now!
Re:Jev Meets the Trolley Problem by nico on Thursday October 15, 2026 @03:24PM (#50108919)
Awesome! Super curious about how it fails, and then figuring out how to over fit, err I mean improve it, haha
Re:Jev Meets the Trolley Problem by zlandes86 on Thursday October 15, 2026 @05:48PM (#50110830)
Okay, I tried it! It doesn't have a zero-shot mode, so the only way I could test it without training was with the SNLI head as judge. On all 5,240 prompts that stand-in ignored the number of people (pulling in about half the wordings whether 0 or 20 people were at stake) and contradicted itself 96% of the time. You could train a "trolley head" but that would just measure whoever labelled the training data!
Re:Jev Meets the Trolley Problem by nico on Thursday October 15, 2026 @09:16PM (#50112520)
> You could train a "trolley head" but that would just measure whoever labelled the training data!

Thank you for giving it a go. Will have to put some time into how to got about it

You are correct though, and it’s an interesting property of training classifiers, that you are essentially encoding the preference of the data labeler(s)