News for hackers. Stuff that matters.
Re:Things we learned about LLMs in 2024 by ninth_ant on Tuesday December 31, 2024 @02:59PM (#42561349)
I think a lot of the confusion is in how we approach LLMs. Perhaps stemming from the over-broad term “AI”.

There are certain classes of problems that LLMs are good at. Accurately regurgitating all accumulated world knowledge ever is not one, so don’t ask a language model to diagnose your medical condition or choose a political candidate.

But do ask them to perform suitable tasks for a language model! Every day by automation I feed in the hourly weather forecast my home ollama server and it builds me a nice readable concise weather report. It’s super cool!

There are lots of cases like this where you can give an LLM reliable data and ask it to do a language related task and it will do an excellent job of it.

If nothing else it’s an extremely useful computer-human interface.

Re:Things we learned about LLMs in 2024 by rrix2 on Tuesday December 31, 2024 @03:09PM (#42561420)
> Every day by automation I feed in the hourly weather forecast my home ollama server and it builds me a nice readable concise weather report.

not to dissuade you from a thing you find useful but are you aware that the national weather service produces an Area Forecast Discussion product in each local NWS office daily or more often that accomplishes this with human meteorologists and clickable jargon glossary?

https://forecast.weather.gov/product.php?site=SEW&issuedby=S...

Re:Things we learned about LLMs in 2024 by ninth_ant on Tuesday December 31, 2024 @05:17PM (#42562266)
Doesn’t dissuade me at all, that’s a really neat service. I’m not American though, and even if my own country had a similar service I still enjoying tuning the results to focus on what I’m interested in. And it was just an example of the kinds of computer-human interfaces that are newly possible from this technology.

Anytime you have data and want it explained in a casual way — and it’s not mission critical to be extremely precise — LLMs are going to be a good option to consider.

More useful AGI-like behaviours may be enabled by combining LLMs with other technologies down the line, but we shouldn’t try to pretend that LLMs can do everything nor are they useless.

Re:Things we learned about LLMs in 2024 by LtWorf on Wednesday January 01, 2025 @02:42AM (#42564571)
The best forecast available on the internet is norwegian.
Re:Things we learned about LLMs in 2024 by pixl97 on Tuesday December 31, 2024 @05:12PM (#42562236)
>don’t ask a language model to diagnose your medical condition

Honestly they are very decent at it if you give them accurate information in which to make the diagnosis. The typical problem people have is being unable to feed accurate information to the model. They'll cut out parts they don't want to think about or not put full test results in for consideration.

Re:Things we learned about LLMs in 2024 by LtWorf on Wednesday January 01, 2025 @02:43AM (#42564576)
Yes so basically bias it into what you think it should reply in the question and it will magically somehow give the reply you wanted! Very useful :D
Re:Things we learned about LLMs in 2024 by ninth_ant on Wednesday January 01, 2025 @12:51PM (#42567679)
If the LLM is trained on accurate medical data and you provide accurate symptoms data, then the LLM can be a useful tool to output the information in a human-readable way.

This is not a diagnosis. Any reasonably capable person can read webmd and apply the symptoms listed and compare them to what the patient describes. This is widely regarded as dangerous because the input data as well as the patient data are limited in ways that can be medically relevant.

So even if you can use it as a good substitute for browsing webmd, it’s still not a substitute for seeing a medical professional. And for the foreseeable future it will not be.

Re:Things we learned about LLMs in 2024 by capr on Monday January 06, 2025 @06:07AM (#42609584)
compared to what? most doctors never heard of Bayes rule
Re:Things we learned about LLMs in 2024 by sdesol on Tuesday December 31, 2024 @06:59PM (#42562813)
> Perhaps stemming from the over-broad term “AI”.

No, I think if we follow the money, we will find the problem.

Re:Things we learned about LLMs in 2024 by dinosaurdynasty on Tuesday December 31, 2024 @07:01PM (#42562827)
> choose a political candidate

I actually found 4o+search to be really good at this... Admittedly what I did was more "research these candidates, tell me anything newsworthy, pros/cons, etc" (much longer prompt) and well, it was way faster/patient at finding sources than I ever would've been, telling me things I never would've figured out with <5 minutes of googling each set of candidates (which is what I've done before).

Honestly my big rule for what LLMs are good at is stuff like "hard/tedious/annoying to do, easy to verify" and maybe a little more than that. (I think after using a model for a while you can get a "feel" for when it's likely BSing.)

Re:Things we learned about LLMs in 2024 by pella on Tuesday December 31, 2024 @07:20PM (#42562912)
> so don’t ask a language model to diagnose your medical condition

(o1-preview) LLMs show promise in clinical reasoning but fall short in probabilistic tasks, underscoring why AI shouldn't replace doctors for diagnosis just yet.

"Superhuman performance of a large language model on the reasoning tasks of a physician" https://arxiv.org/abs/2412.10849 [14 Dec 2024]

Re:Things we learned about LLMs in 2024 by mvdtnz on Tuesday December 31, 2024 @10:03PM (#42563626)
> Every day by automation I feed in the hourly weather forecast my home ollama server and it builds me a nice readable concise weather report. It’s super cool!

You feed it a weather report and it responds with a weather report? How is that useful?

Re:Things we learned about LLMs in 2024 by hansvm on Wednesday January 01, 2025 @12:41AM (#42564202)
It distilled bulk information into a form the author cared about. If nothing else it was probably fun, and a personal report on the things you care about can save minutes each day.

I did something similar awhile back without LLMs. I enjoy kayaking, but for a variety of reasons [0] it's usually unwieldy to break out of the surf and actually get out into the ocean at my local beach. I eventually started feeding the data into an old-school ML model where I'd manually check the ocean and report on a few factors (breaking waves, unsafe wind magnitude/direction, ...). The model converted those weather/tide reports into signals I cared about, and then my forecast could simply AND all those together and plot them on a calendar.

An LLM is less custom in some sense, but if you have certain routines you care about (e.g., commuting to my last job I'd always avoid the 101 in favor of 280 if there was heavy rain), it's easy to let the computer translate raw weather information into signals you care about (should you take an alternate route, should you alter your schedule, ...).

Off-topic, do you know of a good source of weather covariates? E.g., a report with a 50% chance of rain for 2hr can easily mean light rain guaranteed for 2hr, a guaranteed 1hr of rain sometime in that 2hr period, a 50% chance that a 2hr storm will hit your town or the next town over, or all kinds of things. Does anybody report those raw model outputs?

[0] There isn't any protection from the open ocean (combined with a kayak that's a bit too top-heavy for the task at hand), which doesn't help, but the big problem is a sand bar just off the coast. If the tide isn't just right, even small swells are amplified into large breaking waves, and I don't particularly mind getting dumped upside down onto a sand bar, but I'd really prefer to spend that time in slightly calmer waters.

Re:Things we learned about LLMs in 2024 by ninth_ant on Wednesday January 01, 2025 @01:01PM (#42567757)
Well said, that’s exactly what I meant.