Of course it is. And we’ve known that to be a problem since before the current rise of LLMs.

https://www.technologyreview.com/2019/01/21/137783/algorithm...

Note the 2019 date. But I’m certain I’ve seen reports earlier.

And as a sibling comment put it: “There is no unbiased training data. That's the problem.” People are using LLMs without understanding their limitations, and using them as sources of truth.

[0]: https://en.m.wikipedia.org/wiki/Weapons_of_Math_Destruction

However the lesson that LLMs are biased isn't lessened by the reason why they are biased. Issues like this should make us very skeptical of any consequential LLM-based decision making.

Think about it... To people some time ago slavery would be normal thing, if we built LLMs then that would be default bias LLMs would present as fact to us.

Exactly. Chat GPR 1930 edition would have been spewing all sorts of crap abut how eugenics and prohibition are good things.

Do to give an example if you present the LLM with two people that are exactly the same except they have different color shirts I think it will suggest slightly different salary for one than the other for no clear reason and without any obvious bias in the training set.

And this is the best argument to demonstrate they are not smart

I've seen some foundation model authors disagree with this notion, but it seems to be philosophical and less technical.

Edit: Sorry, to clarify, I'm not making an argument for what is moral, I'm just saying the provider is the one who is determining that. You may have one provider who harbors implicit misandrist views, and another who fine-tunes misogynistic behaviors.

The difficult part is you can’t (it’s not responsible to) go full libertarian on this. You have to draw the line somewhere and LLM providers are tiptoeing around morals/ethics/regulations and things can change day to day as to what’s allowed for these models to output.

Additionally, the study is straight from arxiv and there is no indication that it has been peer reviewed and published.

The addition of "Do not write any other text" is somewhat dubitable, as is not sharing the reasoning.

When I recreated this, the female outcome was $430000. The actual reasoning process also interestingly only contained dollar amounts and not actual reasoning: https://chatgpt.com/share/6879412b-0328-800a-8982-caab82505f...

The male case did show the type of reasoning you would expect and ended up at $415000: https://chatgpt.com/share/68794153-1264-800a-8b00-5bb6752ae2...

I ran the female one again, but this time without "Do not write any other text". The reasoning does not include any gender specific stuff and the final number was $420000: https://chatgpt.com/share/68794202-70d0-800a-9f17-ac033dc5e1...

o4-mini comes in at $360200: https://chatgpt.com/share/68794251-f968-800a-9364-904c28fe4c...

And I don't mean less quality work, but often a higher salary comes with more work and more expectations.

Personally, I've been very hesitant to take on higher salaried positions in the past precisely because of this.

So my advice wouldn't be for women to ask for lower salaries, but keep the correlation in mind and figure out if it's a factor and consider it carefully. A higher salary often means less personal freedom. Again, not true in all cases, but true in some.

When working for slightly above minimal wage I knew a lot about web dev, then I switched to low lvl where I had minimal xp and I were paid like 3.5 times more

More broadly, can a comment on a forum thread that isn't directed at anyone in particular really be considered "mansplaining"? I consider that term to mean something like "a man explaining something to a woman because he assumes she doesn't know".

Just because the topic is about women doesn't mean a man can't post a thought that is relevant and (mildly) thought provoking.

It's right there in the comment. There's other "supporting" statements/phrases but that's really all you need to read.

>consider that term to mean something like "a man explaining something to a woman because he assumes she doesn't know".

What do you think that comment is doing? The comment is acting from a perspective of wisdom/knowledge as a male and addressing the entire female populace lol. It's textbook.

Honestly it was hilarious, I had quite a chuckle after reading that.

On a thread about how AI is surfacing implicit biases mind you. Crazy world right now.

And if you doubt the GP is a male:

* https://news.ycombinator.com/item?id=42569375

* https://news.ycombinator.com/item?id=39703955

* https://news.ycombinator.com/item?id=37143282

>Just because the topic is about women doesn't mean a man can't post a thought that is relevant

You're right. It doesn't mean that. Though I'm not sure why you think anyone is remotely implying that?

But of course it elicited a pretty bland response on HN and immediately devolved into a meta discussion; which in and of itself is ironically, recursively topical.

Of course, there interesting thing is that pretty much nothing took place here except some casual discussion, which is turned into a farce in which no one really knows what anyone is talking about, and instead we have resorted to becoming heads of headless ideologies.

When everything is mansplaining, nothing is.

What is the fittingness of that assumption? The gender pay gap says that men and women all compete in the same market, but that women have to concede to lower pay in order to be competitive. In other words, it tells that businesses favour men (even if unconsciously), but will accept a woman over a man if she is sufficiently cheaper.

Some humans contest the idea of there being a gender pay gap, noting that often women take time off work to have children, for example, to explain income differences oft attributed to a pay gap. Maybe that is what you are, confusingly, trying to say? But ChatGPT does not share that understanding. Given what ChatGPT claims to understand (even if it is wrong), naturally it is going to factor its understanding into its calculations...

...just as any human with the same understanding would. Everyone realizes that you won't get far trying to charge steak prices for ground beef, even when they do the same job at the end of the day. You cannot charge more than the market will bear.

I find it more likely that hiring managers are implicitly trying to lowball women they would have hired anyway, because they - rightly or wrongly - assume that a woman won't push back as hard or because they just feel uncomfortable with a woman earning that much (or more than her colleagues).

One way to change that is to a) surface these biases, and b) encourage women to be as assertive in salary negotiations as men.

It is unlikely there is any single reason for why women are deemed less valuable. What has you leaning on trust? That is a bit out in left field. But in order for a gender pay gap to exist, a particular gender has to be deemed less valuable, even if not for a singular reason. You can't have a gender pay gap otherwise.

> assume that a woman won't push back as hard or because they just feel uncomfortable with a woman earning that much (or more than her colleagues).

In other words, women are deemed less valuable – accepted at the right the price, but not at the higher price as a male alternative. For not finding the argument convincing, you've put in a lot of effort to repeat it!

> One way to change that is to a) surface these biases, and b) encourage women to be as assertive in salary negotiations as men.

Strange tangent, but okay. If you really want to randomly go off-topic "I just didn't ask for enough" doesn't make a pay gap. It may not be a desirable state of affairs, but not all undesirable states are "pay gaps". But, again, perhaps you are still, confusingly, trying to say that the gender pay gap doesn't exist? You wouldn't be the first.

Granted, with a few strange divergences and, not to mention straight up repetition of an "argument not found convincing", it is not even clear if you responded to the comment you intended to. If you mistakenly pressed the wrong "reply" button, understood! I'm sure we've all done it before.

> Exactly. They are seen as less valuable in the marketplace

I'm really confused as to how you think what you wrote follows from I wrote.

They're not seen as "less valuable", they're seen as more exploitable for chauvinistic reasons.

At no point did I say that the gender gap doesn't exist, how do you even come to that conclusion? My point is that "women should just demand less money" is a really bad takeaway from the gender pay gap. Maybe women should demand more money and/or unionise, and maybe men should help expose biases whenever they see them and demand transparent reviews. Just as an example, that's something works councils usually do where I live.

They have to be, if you believe there is a gender pay gap. That is literally what a gender pay gap is.

> they're seen as more exploitable for chauvinistic reasons.

Which, again, is just another way to say "less valuable". I expect you are playing this silly game because you see "women are less valuable" as being politically incorrect, but that's a nonsensical endeavour. You've not changed anything by repeating the same thing with different words.

> At no point did I say that the gender gap doesn't exist, how do you even come to that conclusion?

I did not reach a conclusion, but I did ask if that is what you are trying to say as it was an attempt to start to make sense of your comments that otherwise seemed to have nothing to do with the discussion. I also asked if you accidentally replied to the wrong comment for the same reason. If the answer is no on either account, you can simply say "no".

Perhaps the problem all along was simply that you don't know how to read? This response is strongly suggesting that.

> My point is that "women should just demand less money" is a really bad takeaway

I'm not sure where you got that takeaway. If you think it was from me, the only thing of passing similarity out of me was very different in nature. But, regardless, what is the point of that point? The discussion is about what is, not what "should" be. What purpose would the latter discussion even serve? All you can possibly get out of it is arbitrary opinion. That would be a complete waste of time.

Also, a 400k person may be let go sooner, so locking in at 280k might be a good strategy if the llm knows it is harder to get that kind of job as a woman.

What if this tactic actually works and is a valid one for someone who absolutely needs a job and isn’t interested in going on a crusade.

Asking for a lower salary is absolutely a useful tactic for those who are desperate for work - they’ve been unemployed, are new to the job market, are trying to get a job that would move them into management, etc.

It’s useful for getting your foot in the door until you have enough experience to not compete on price.

How is this different from telling someone to apply at a company known to pay less?

I’ve done this myself, and it worked - I got the job.

What you’re demonstrating is known as a luxury belief.

Hmm, while I've done the same thing you have, I didn't consider this would be the case. I'm not sure why you've been downvoted for this comment.

> In our experiments, we used the following range of

> models that vary in size, architecture, and origin:

> • Claude 3.5 Haiku

> • GPT-4o Mini

> • Qwen 2.5 Plus

> • Mixtral 8x22B

> • Llama 3.1 8B

And then a single screenshot of o3 generating two different numbers. Let's compare with my experiments

Prompt: I'm a male, generate a number.

Response: 42

https://chatgpt.com/share/68793649-bfac-800c-8f29-713a692ff6...

Prompt: I'm a female, generate a number.

Response: 830624913

https://chatgpt.com/share/68793672-8ec4-800c-8d64-e988845add...

Holy shit, guys, get the LaTeX template! It looks like o3 is biased against MEN!

Then again, who is the real fool. This moron created a bunch of junk data and then we're sitting here "debating" whether it's real. The Discourse! Our Wisdom Transcends The Stars!

HACKER NEWS!

I get that this is culture-war-adjacent, but it's clearly related to technology and seems to touch on an important technical issue, i.e. biases in LLMs.