News for hackers. Stuff that matters.
Re:The Behavior of LLMs in Hiring Decisions: Systemic Biases... by yahoozoo on Tuesday May 20, 2025 @08:15AM (#44040686)
I am skeptical whenever I see someone asking a LLM to include some kind of numerical rating or probability in its output. LLMs can’t actually _do_ that, it’s just some random but likely number pulled from its training set.

We all know the “how many Rs in strawberry” but even at the word level, it’s simple to throw them off. I asked ChatGPT the following question:

> How many times does the word “blue” appear in the following sentence: “The sky was blue and my blue was blue.”

And it said 4.

Re:The Behavior of LLMs in Hiring Decisions: Systemic Biases... by atworkc on Tuesday May 20, 2025 @08:25AM (#44040778)
```

Attach a probability for the answer you give for this e.g. (Answer: x , Probability: x%)

Question: How many times does the word “blue” appear in the following sentence: “The sky was blue and my blue was blue.”

```

Quite accurate with this prompt that makes it attach a probability, probably even more accurate if the probability is prompted first.

Re:The Behavior of LLMs in Hiring Decisions: Systemic Biases... by brookst on Tuesday May 20, 2025 @08:31AM (#44040832)
LLMs can absolutely score things. They are bad at counting letters and words because the way tokenization works; “blue” will not necessarily be represented by the same tokens each time.

But that is a totally different problem from “rate how red each of these fruits are on a scale of 1 (not red) to 5 (very red): tangerine, lemon, raspberry, lime”.

LLMs get used to score LLM responses for evals at scales and it works great. Each individual answer is fallible (like humans), but aggregate scores track desired outcomes.

It’s a mistake to get hung up on the meta issue if counting tokens rather than the semantic layer. Might as well ask a human what percent of your test sentence is mainly over 700hz, and then declare humans can’t hear language.

Re:The Behavior of LLMs in Hiring Decisions: Systemic Biases... by sabas123 on Tuesday May 20, 2025 @08:41AM (#44040920)
I asked ChatGPT, gemini and both answered 3, with various levels of explainations. Was this a long time ago by any chance?
Re:The Behavior of LLMs in Hiring Decisions: Systemic Biases... by xigency on Tuesday May 20, 2025 @02:17PM (#44044344)
67% of the time it works all the time?
Re:The Behavior of LLMs in Hiring Decisions: Systemic Biases... by fastball on Tuesday May 20, 2025 @09:26AM (#44041371)
Sure if you ask them to one-shot it with no other tools available.

But LLMs can write code. Which also means they can write code to perform a statistical analysis.