Re:LLM leaderboard focusing on assessing their biases
by mewpmewp2 on Sunday March 24, 2024 @08:13AM
(#39806727)
Also the probability questions, I don't know how they would interpret for example answers of whether a woman or man would have higher chance of being a Tech CEO, because if you answer equal you are essentially denying that there's any inequality between the genders.
[ Parent
| on HN ]
Re:LLM leaderboard focusing on assessing their biases
by softmodeling on Sunday March 24, 2024 @09:45AM
(#39807187)
It also depends on how/where the LLM is going to be used. If you're using, let's say, an LLM in hiring selection process, you want in fact to be sure that the LLM does consider genders equal as it would be illegal to discriminate based on gender
Re:LLM leaderboard focusing on assessing their biases
by mewpmewp2 on Sunday March 24, 2024 @07:39PM
(#39811528)
Yeah, but you should never word a question like that to an LLM.
Or actually have your bias testing prompt dataset list out person's qualifications and add race, gender there as well. Then do a comparison if LLM scores the fit differently depending on race/gender. This would be much more practical.