News for hackers. Stuff that matters.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by echelon on Saturday December 28, 2024 @12:54PM (#42532859)
I get very wrong and dangerous answers from AI frequently.

I just searched "what's the ld50 of caffeine" and it says:

> 367.7 mg/kg bw

This is the ld50 of rats from this paper: https://pubmed.ncbi.nlm.nih.gov/27461039/

This is higher than the ld50 estimated for humans: https://en.wikipedia.org/wiki/Caffeinism

> The LD50 of caffeine in humans is dependent on individual sensitivity, but is estimated to be 150–200 milligrams per kilogram of body mass (75–100 cups of coffee for a 70 kilogram adult).

Good stuff, Google.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by beejiu on Saturday December 28, 2024 @01:18PM (#42533100)
Perhaps a more common question: "How many calories do men need to lose weight?"

Google AI responded: "To lose weight, men typically need to reduce their daily calorie intake by 1,500 - 1,800 calories"

Which is obviously dangerous advice.

IMO Google AI overviews should not show up for anything (a) medical or (b) numerical. LLMs just aren't safe enough yet.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hinkley on Saturday December 28, 2024 @03:38PM (#42534450)
If you’re aiming to lose weight safely the rule of thumb is 3 lbs a week. 3000kcal per pound works out to an average deficit of about 1280 calories per day. Max.

Obese people can lose a bit more under doctor supervision. My understanding is that it’s tied partially to % of body weight lost per week and partly to what your organs can process, which does not increase with body mass.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by layer8 on Saturday December 28, 2024 @06:14PM (#42535706)
I don’t think absolute numbers are very useful here. You need around 5–10% reduction in calorie intake to get any weight-loss effect going, and I wouldn’t reduce by more than 20% (relative to weight-maintaining intake — it’s different if you’ve been seriously overeating) if you want it to be sustainable longer-term.

So for example if your weight is stable at 2500 kcal per day, I would start by reducing the intake by 250–500 kcal, but not more. If this works well for a month or two and then you want to lose weight faster, you can still reduce your intake further. You generally have to do that anyway even just to maintain the velocity, because weight loss also tends to reduce calorie expenditure.

First and foremost, you need to monitor your calorie intake against weight. Here is a useful text about that: https://www.fourmilab.ch/hackdiet/

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hinkley on Saturday December 28, 2024 @07:44PM (#42536325)
Your body will get more efficient at whatever exercise you do to make the calories work out. So over time you’ll either have to increase your exercise or rein in the calories a bit more to achieve a sustained result.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by layer8 on Sunday December 29, 2024 @10:23AM (#42540459)
That is assuming that you do any exercise. But yes, and the method I linked to explains how to handle any such variation in calorie expenditure regardless of its cause.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by warner25 on Saturday December 28, 2024 @01:29PM (#42533206)
I think even when the answer is "right" in some sense, it should probably come within the context of a bunch of caveats, explanations, etc.

But maybe I'm just weird. Oftentimes when my wife or kids ask me a question, I take a deep breath and start to say something like "I know what you're asking, but there's not a simple or straightforward answer; it's important to first understand ____ or define ____..." by which time they get frustrated with me.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hombre_fatal on Saturday December 28, 2024 @01:49PM (#42533394)
> I think even when the answer is "right" in some sense, it should probably come within the context of a bunch of caveats, explanations, etc.

Funnily enough, this is exactly what the LLM does with these questions. So well that people usually try to tweak their prompts so they don't have to wade through additional info, context, hedging, and caveats.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by layer8 on Saturday December 28, 2024 @05:58PM (#42535586)
So you are saying that Google should provide responses that are more likely to frustrate its users? ;)
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hombre_fatal on Saturday December 28, 2024 @01:41PM (#42533328)
I don't see the problem with the answer, and the question is already garbage. Plus, the LLM hedges its advice with precautions.

I get a pretty good summary when I paste the question into Google. It comes up with a ballpark but also gives precautions and info on how to estimate what caloric restriction makes sense for you within the first 3 sentences.

And all in a format someone is likely to read instead of clicking on some verbose search result that only answers the question if they read a whole article which they aren't going to do.

This seems like really lame nit picking. And I don't think it passes the "compared to what?" test.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by Jaxan on Saturday December 28, 2024 @01:55PM (#42533447)
The question is garbage. But people will ask it with their best intentions and not know it’s garbage.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by beejiu on Saturday December 28, 2024 @02:03PM (#42533522)
The basic problem is it says reduce "by 1,500 - 1,800" rather than "to 1,500 - 1,800" (not that that answer is much better). Yes, it's a garbage question, but the first answer is unsafe in all circumstances. The simplest solution here is to show nothing.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by griomnib on Saturday December 28, 2024 @02:10PM (#42533584)
Funny thing is you can train a small BERT model to detect queries that are in categories that aren’t ready for AI “answers” with like .00000001% of the energy of an LLM.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hansvm on Saturday December 28, 2024 @03:49PM (#42534590)
That's (obviously) a bit of an exaggeration. BERT is just another transformer architecture. Cut down from ~100 layers to 1, ~1k dimensions to ~10, and ~10k tokens to 100, and you're only 1e6 faster / more efficient, still a factor of 10k greater than your estimate and also too small to handle the detection you're describing with any reasonable degree of accuracy.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by griomnib on Saturday December 28, 2024 @05:17PM (#42535292)
I literally have DistilBERT models that can do this exact task in ~14ms on an NVIDIA A6000. I don’t know the precise performance per watt, but it’s really fucking low.

I use LLM to help with training data as they are great at zero shot, but after the training corpora is built a small, well trained, model will smoke an LLM in classification accuracy and are way faster - which means you can get scale and low carbon cost.

In my personal opinion there is a moral imperative to use the most efficient models possible at every step in a system design. LLM are one type of architecture and while they do a lot well, you can use a variety of energy efficient techniques to do discrete tasks much better.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hansvm on Saturday December 28, 2024 @05:53PM (#42535552)
Thanks for providing a concrete model to work with. Compared to GPT3.5, the number you're looking for is ~0.04%. I pointed out the napkin math because 0.00000001% was so obviously wrong even at a glance that it was hurting your claim.

And, yes, purpose-built models definitely have their place even with the advent of LLMs. I'm happy to see more people working on that sort of thing.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by griomnib on Saturday December 28, 2024 @06:53PM (#42536010)
I applaud you doing the math! Proves you aren’t an LLM :-D
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by nuancebydefault on Saturday December 28, 2024 @02:22PM (#42533680)
If your calorie intake is just 1500 today, it is bad advice. If your calorie surplus is 1800, it is good advice.

But I wonder, were those few words the full response? Information hiding to prove a point is too easy.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hansvm on Saturday December 28, 2024 @03:54PM (#42534635)
A calorie surplus of 1800/day is ~190 lbs/yr. Is that something people actually do?
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by tredre3 on Saturday December 28, 2024 @06:58PM (#42536048)
Yes. You can't just say that someone eating 1800 over the recommended 2000 will perpetually gain weight. Weight maintenance calories will depend on the weight of the person.

A 500lbs man will need to consume 4000kcals/day to not lose weight. Cutting 1800 of that is realistic and might be good advice on the LLM's part, so it really depends on how GP asked the question.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by sroussey on Saturday December 28, 2024 @01:41PM (#42533326)
> Which is obviously dangerous advice.

Same advice as my trainer gives me.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by Retric on Saturday December 28, 2024 @01:59PM (#42533486)
Did they say:

  reduce their daily calorie intake to 1,500 - 1,800 calories
  or
  reduce their daily calorie intake by 1,500 - 1,800 calories
These are very different answers, unless you’re consuming ~3,300 calories per day. These kinds of ‘subtle’ phrasing issue often results in AI mistake as both words are commonly used in advice but the context is really important.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by sroussey on Saturday December 28, 2024 @02:26PM (#42533719)
Oh yeah! No, reduce to not reduce by. Though at the time I was eating a few things that had high calories that I didn’t realize so it would have been the same.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by simoncion on Saturday December 28, 2024 @01:54PM (#42533436)
Your trainer advises you to reduce your calorie intake to between 200 and 500 calories per day? [0] That sounds very, very hazardous for anything other than very short term use, and (given the body's inbuilt "starvation mode") probably counterproductive, even then.

[0] Note that the robot suggested to reduce calorie intake by 1,500->1,800 calories, and the recommended calorie intake is 2,000.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hinkley on Saturday December 28, 2024 @03:44PM (#42534522)
People losing weight are probably eating more than 2000 per day to begin with. But if you go from 2800 down to 1500 you’re already likely to exceed 3 lbs of weight loss per week that is recommended without doctor supervision. If you need to lose more than 150 lbs in a year because you’re well past morbid obesity then you need staff, not just a food plan.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by sroussey on Sunday December 29, 2024 @11:50AM (#42541059)
If you’re eating out at Chilies, you could easily be eating 3000 calories per meal.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hinkley on Sunday December 29, 2024 @04:21PM (#42543287)
I recall when Krispy Kreme came out with a donut shake that was 1800 calories for the large size. It’s crazy out there.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by zaroth on Saturday December 28, 2024 @02:33PM (#42533784)
It’s not even advice, and it’s not wrong.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by trollbridge on Saturday December 28, 2024 @01:56PM (#42533453)
That explains why I haven't been losing weight!
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by Aeolun on Saturday December 28, 2024 @02:01PM (#42533505)
I think that would explain why you’re starving, not how you’re not losing weight.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by chimpanzee on Saturday December 28, 2024 @01:02PM (#42532936)
For whatever it’s worth, in response to the same question posed by me (“what is the ld50 of caffeine”), Google’s AI properly reported it as 150-200 mg/kg.

I asked this about 1 minute after you posted your comment. Perhaps it learned of and corrected its mistake in that short span of time, perhaps it reports differently on every occasion, or perhaps it thought you were a rat :)

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by WarOnPrivacy on Saturday December 28, 2024 @01:33PM (#42533247)

    The median lethal dose (LD50) of caffeine in humans is estimated to be 150–200 milligrams per kilogram of body mass. However, the lethal dose can vary depending on a person's sensitivity to caffeine, and can be as low as 57 milligrams per kilogram. 

    Route of administration 
    Oral 367.7 mg/kg bw
    Dermal >2000 mg/kg bw
    Inhalation LC50 combined: ca. 4.94 mg/L
ref: https://i.imghippo.com/files/yeKK3113pE.png 13:25EST (by a Kagi shill ftr)
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hinkley on Saturday December 28, 2024 @03:49PM (#42534581)
That’s the danger with thinking in terms of LD50.

That’s half the people in a caffeine chugging contest falling over dead. The first 911 call would be much much earlier. I doubt you’d get to 57 mg before someone thought they were having a heart attack (angina).

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by raincole on Saturday December 28, 2024 @01:20PM (#42533112)
Perhaps Google AI reads HN at work just like us.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by pests on Saturday December 28, 2024 @01:04PM (#42532957)
I also got similar and just tried, we are posting within minutes.

--

The median lethal dose (LD50) of caffeine in humans is estimated to be 150–200 milligrams per kilogram of body mass. However, the lethal dose can vary depending on a person's sensitivity to caffeine, and can be as low as 57 milligrams per kilogram. Route of administration LD50 Oral 367.7 mg/kg bw Dermal 2000 mg/kg bw Inhalation LC50 combined: ca. 4.94 mg/L The FDA estimates that toxic effects, such as seizures, can occur after consuming around 1,200 milligrams of caffeine.

There was a table in the middle there.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by griomnib on Saturday December 28, 2024 @02:11PM (#42533597)
LLM are non deterministic by nature.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by chimpanzee on Saturday December 28, 2024 @02:30PM (#42533758)
Yes, that’s the main issue as ideally they wouldn’t be non-deterministic on well-established quantitative facts.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by griomnib on Saturday December 28, 2024 @02:37PM (#42533819)
But they can never be. RAG gets you somewhere, but it’s still a pile of RNGs under a trenchcoat.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by chimpanzee on Saturday December 28, 2024 @03:17PM (#42534228)
>> ideally
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by griomnib on Saturday December 28, 2024 @04:57PM (#42535158)
It’s just not possible. You can do a lot with nondeterministic systems, they have value - but oranges and apples. They need to coexist.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by chimpanzee on Saturday December 28, 2024 @05:42PM (#42535486)
ideal (def. #2) = Existing only in the mind; conceptual, imaginary

https://en.m.wiktionary.org/wiki/ideal

(We’re allowed to imagine the impossible.)

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by griomnib on Saturday December 28, 2024 @06:55PM (#42536027)
Fair, I am loath to take away your dreams!
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by richk449 on Saturday December 28, 2024 @04:24PM (#42534882)
Is this really true? The linear algebra is deterministic, although maybe there is some chaotic behavior with floating point handling. The non deterministic part mostly comes from intentionally added randomness, which can be turned off right?

Maybe the argument is that if you turn off the randomness you don’t have an LLM like result any more?

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hansvm on Sunday December 29, 2024 @01:09AM (#42537997)
Floats are deterministic too (this winds up being helpful if you want to do something like test an algorithm on every single float); you just might get different deterministic outcomes on different compilation targets or with threaded intermediate values.

The argument is, as you suggest, that without randomness you don't have an LLM-like result any more. You _can_ use the most likely token every time, or beam search, or any number of other strategies to try to tease out an answer. Doing so gives you a completely different result distribution, and it's not even guaranteed to give a "likely" output (imagine, e.g., a string of tokens that are all 10% likely for any greedy choice, vs a different string where the first is 9% and the remainder are 90% -- with a 10-token answer the second option is 387 million times more likely with random sampling but will never happen with a simple deterministic strategy, and you can tweak the example slightly to keep beam search and similar from finding good results).

That brings up an interesting UI/UX question.

Suppose (as a simplified example) that you have a simple yes/no question and only know the answer probabilistically, something like "will it rain tomorrow" with an appropriate answer being "yes" 60% of the time and "no" 40%. Do you try to lengthen the answer to include that uncertainty? Do you respond "yes" always? 60% of the time? To 60% of the users and then deterministically for a period of time for each user to prevent flip-flopping answers?

The LD50 question is just a more complicated version of that conundrum. The model isn't quite sure. The question forces its hand a bit in terms of the classes of answers. What should its result distribution be?

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by vidarh on Saturday December 28, 2024 @01:22PM (#42533137)
I get your point wasn't this specific example, it's perhaps not a very good example of being dangerous: Getting that much caffeine into your bloodstream takes quite a commitment, and someone who knows the term LD50 is perhaps not very likely to think it indicates what is safe to consume. It's also not something you're likely to do accidentally because you've looked it up online and decided to test it.

In the most concentrated form in typical commercial caffeine tablets, it's half to one fistful. In high-caffeine pre-workout supplements, it's still a quantity that you'd find almost impossible to get down and keep down... E.g. a large tumbler full of powder of mine with just enough water to make it a thick slurry you'd likely vomit up long before much would make it into your bloodstream...

I'm not saying it's impossible to overdose on caffeinated drinks, because some do, and you can run into health problems before that, but I don't think that error is likely to be very high on the list of dangerous advice.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hinkley on Saturday December 28, 2024 @03:09PM (#42534150)
Even that seems high. I don’t feel good with 200mg per human, not per kilo. I can’t imagine drinking ten times as much and not being in the ER. A hundred times that much? No fucking way.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by alex_young on Saturday December 28, 2024 @01:07PM (#42532985)
Hmm my search returns “between 150 to 200 mg per kilogram”, which is maybe more correct?

Also, in what context is this dangerous? To reach dangerous levels one would have to drink well over 100 cups of coffee in a sitting, something remarkably hard to do.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by dijksterhuis on Saturday December 28, 2024 @01:17PM (#42533076)
> Also, in what context is this dangerous? To reach dangerous levels one would have to drink well over 100 cups of coffee in a sitting

some people use caffeine powder / pills for gym stuff apparently.

someone overdosed and died after incorrectly weighing a bunch of powder.

doubt it is a big leap to someone dying because they were told the wrong limits by google.

https://www.bbc.co.uk/news/uk-wales-60570470

as ever, machine learning is not really suitable for safety/security critical systems / use cases without additional non-ML measures. it hasn’t been in the past, and i’ve seen zero evidence recently to back up any claim that it is.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by vidarh on Saturday December 28, 2024 @01:43PM (#42533341)
I don't doubt the news article on this, but even with caffeine pills/powder it's near half a fistful to get to LD50 judging by my caffeine tablets. It's not impossible to consume, but it'd be distinctly unpleasant long before you get even anywhere close to dangerous levels.

For my high-caffeine pre-workout powder, I suspect I'd vomit long before I'd get anywhere near. Pure caffeine is less unpleasant, but still pretty awful, which I guess is why we don't see more deaths from it despite the widespread use.

I agree with you that there really ought to be caution around giving advice on safety-critical things, but this one really is right up there in freak accident territory, in the intersection of somewhat dangerous substances sold in a poorly regulated form (e.g. there's little reason for these to be sold as bulk powders instead of pressed into pills other than making people feel more macho downing awful tasting drinks instead of taking pills).

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hinkley on Saturday December 28, 2024 @03:22PM (#42534286)
I wonder if they’re thinking 200mg per kilo to trigger cell death. I have trouble believing a human heart surviving a dose of 50mg/kg. Half of them surviving four times that much? No. I don’t believe it.

Found an article about a teenager who died after three strong beverages. The coroner is careful to point out that this was likely an underlying medical condition not the caffeine. The health professional they interviewed claims 10g is lethal for “most” people, which would be 100-150mg/kg. That still seems like something an ER doctor would roll their eyes at.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hombre_fatal on Saturday December 28, 2024 @01:46PM (#42533368)
Your example doesn't interact with the chicken littling in this thread.

> The hearing was told the scales Mr Mansfield had used to measure the powder had a weighing range from two to 5,000 grams, whereas he was attempting to weigh a recommended dose of 60-300mg.

Nothing to do with an LLM nor with someone not knowing the exact LD50 of caffeine. Just "this article contains someone dying of caffeine overdose, and we're talking about caffeine overdose here, therefore LLM is dangerous."

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hinkley on Saturday December 28, 2024 @03:11PM (#42534166)
This is why we let the pros do compounding. Slip a decimal point and you can kill yourself with many substances.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by TeMPOraL on Saturday December 28, 2024 @01:28PM (#42533202)
> some people use caffeine powder / pills for gym stuff apparently.

At 200mg per pill, which is the strongest I had, I'd still have to down some 70+ pills in one go. Not strictly impossible, but not something you could possibly do by accident, and even for the purpose of early check-out, it wouldn't be my first choice.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by vidarh on Saturday December 28, 2024 @01:46PM (#42533372)
An accident with it in powdered form is possible - people who use them are often used to pre-workout supplements tasting awful, and so might be prepared to down it as fast as possible - but it's a big enough volume of powder that it really is a freak accident.

And if on purpose, using caffeine would just be staggeringly awful...

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by dijksterhuis on Saturday December 28, 2024 @01:43PM (#42533351)
the problem isn’t someone’s intent (on purpose/by accident).

it’s intent (want to improve my gym performance so down a bunch of caffeine) combined with incorrect information gained from what is supposedly a trustworthy source (the limit presented is much higher than it actually is for humans).

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by vidarh on Saturday December 28, 2024 @01:48PM (#42533390)
If they're searching for LD50, they're already setting themselves up for errors, even with the right information. The LD50 isn't a safe dose, after all, but the mean lethal dose. While it's not great if its wrong, if people search for an LD50 thinking it indicates what they can safely take, it's already going to be hard to protect them against themselves.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by nmeagent on Saturday December 28, 2024 @02:18PM (#42533653)
Yes, Google's AI chatbot confidently claimed yesterday that US passports have a fingerprinting requirement, which is absolutely not true. These things can't be trusted to emit even basic facts without somehow screwing it up and it's frankly depressing how they are worming their way into almost everything. I hope this particular hype train is derailed as soon as possible.