LLM leaderboard focusing on assessing their biases

Re:LLM leaderboard focusing on assessing their biases by softmodeling on Sunday March 24, 2024 @09:42AM (#39807176)

You can configure the "communities" you want to test to make sure the LLM doesn't have biases against any of them (or, depending on the type of prompt, that the LLM offers the same answer regardless the community you use in the prompt, i.e. that the answers doesn't change when you replace "men" by "women" or "white" by "black")

[ Parent | on HN ]

Re:LLM leaderboard focusing on assessing their biases by maxcoder4 on Sunday March 24, 2024 @10:21AM (#39807390)

"Is [black] people skin usually dark?"

"Can a majority of [women] become pregnant?"

I don't see how one can expect the same answer when substituting variables for various genders, races and social classes, and still expect the same responses. But I'm still trying to understand the methodology, I'm sure it's more complex than that.

[ Parent | on HN ]

Re:LLM leaderboard focusing on assessing their biases by softmodeling on Sunday March 24, 2024 @10:30AM (#39807457)

Well, indeed, the parameters make sense for the templates provided. Not for any type of question

[ Parent | on HN ]

Re:LLM leaderboard focusing on assessing their biases by maxcoder4 on Sunday March 24, 2024 @11:27AM (#39807826)

But do they? For example there are much more female nurses than male nurses. I don't understand the point of asking for a "probability a (GENDER) has to be a nurse". It's not even clear if the question is about the current status, or about the goal for which we should strive for.

[ Parent | on HN ]