News for hackers. Stuff that matters.
Re:MIT researchers use large language models to flag problem... by glutamate on Thursday August 15, 2024 @04:10AM (#41253986)
Some people, when confronted with a complex system, think "I know, I'll use a large language model." Now they have two complex systems interacting.
Re:MIT researchers use large language models to flag problem... by cqqxo4zV46cp on Thursday August 15, 2024 @04:26AM (#41254048)
Some people trivialise things in bad faith to score internet points on Hacker News, I guess. The status quo, per the article, was already using “deep learning” models to perform anomaly detection. We are already talking about complex, black-box systems.

The stated advantage was that they didn’t require deployment-specific training. You can just throw a pre-trained LLM at it. There is also a stated benefit: early-stage detection, without needing to pay for, or wait for, training a custom ML model. The article is quite open about the fact that the LLM approach doesn’t beat the state of the art in terms of accuracy.

It’s like you didn’t click the link.

Re:MIT researchers use large language models to flag problem... by dotnet00 on Thursday August 15, 2024 @07:44AM (#41254893)
>early-stage detection, without needing to pay for, or wait for, training a custom ML model

This doesn't make much sense though, if you're in a hurry to get something working and don't have the data for training a custom model, you'd use the knowledge you have of the system to setup a typical statistical anomaly detector. That way you'd have the insight into the system behavior that you'd want early on, would be able to enforce all sorts of guarantees and would not have the weight and complexity of an LLM on the system's resources, rather than worrying about this LLM that has nowhere near the same info on the system.

Using an LLM for this would be like hiring a random person off the street to keep your network infrastructure working because you're in such a rush that you don't even want to document how your infra is setup, let alone wait to find someone who would know how to decipher it.

Re:MIT researchers use large language models to flag problem... by mdp2021 on Thursday August 15, 2024 @04:49AM (#41254152)
The metrics OP suggest increase with the transition from DL to LLMs (and OP had no immediate duty to discuss other metrics).
Re:MIT researchers use large language models to flag problem... by glutamate on Thursday August 15, 2024 @04:57AM (#41254198)
Colour me old-fashioned, but i am really sceptical about a zero-shot general-purpose LLM-based anomaly detector being able to outperform a trivial, not-obviously-wrong ML/statistical model calibrated on a small subset of the data that makes training extremely fast, in entirely new domains. And in anomaly detection, false negatives (and even false positives) can be very expensive.

The most promising role for LLMs in timeseries prediction is in extracting covariates from unstructured data, in the wind example, weather reports and geographically proximal social media posts.

Re:MIT researchers use large language models to flag problem... by thomashop on Thursday August 15, 2024 @05:02AM (#41254208)
You are skeptical about something that is explained quite thoroughly in the article. Did you read the article?

Edit: I meant they never claim to outperform SOTA domain-specific approaches. So, I don't know whose claim you are refuting.

Re:MIT researchers use large language models to flag problem... by glutamate on Thursday August 15, 2024 @05:06AM (#41254225)
> While LLMs could not beat state-of-the-art deep learning models at anomaly detection, they did perform as well as some other AI approaches.

I think one should also be sceptical about the current hype train

Re:MIT researchers use large language models to flag problem... by Kiro on Thursday August 15, 2024 @06:20AM (#41254510)
Funny how one of the the few comments actually discussing the content of the article gets downvoted while generic memes that have nothing to do with it get upvoted.
Re:MIT researchers use large language models to flag problem... by visarga on Thursday August 15, 2024 @04:33AM (#41254076)
> Now they have two complex systems interacting.

As opposed to sending crowds of people to interact with the system? We are even more complex and less benchmarked/tested than LLMs. At least the LLM is a known thing, we know its limitations, we can evaluate and compensate for their lacks.

Re:MIT researchers use large language models to flag problem... by mensetmanusman on Thursday August 15, 2024 @06:52AM (#41254626)
The human body is orders more complex than LLMs and we can already use LLMs to improve anomaly detection rates for cancer.
Re:MIT researchers use large language models to flag problem... by theGnuMe on Thursday August 15, 2024 @07:10AM (#41254725)
> improve anomaly detection rates for cancer.

What do you mean by that?

Re:MIT researchers use large language models to flag problem... by mensetmanusman on Thursday August 15, 2024 @01:05PM (#41258158)
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11003452/

“We propose and demonstrate the use of these models to assist in the manual curation process which results in higher accuracy and F1 score with lesser time and cost, thus improving efforts of cancer research.”

Re:MIT researchers use large language models to flag problem... by theGnuMe on Thursday August 15, 2024 @01:09PM (#41258199)
How does this relate to anomaly detection?
Re:MIT researchers use large language models to flag problem... by mensetmanusman on Friday August 16, 2024 @02:36PM (#41269077)
Anomalys are markers to keep track of when diagnosing.
Re:MIT researchers use large language models to flag problem... by theGnuMe on Friday August 16, 2024 @03:19PM (#41269467)
That paper doesn't even mention the word anomaly or detecting them.
Re:MIT researchers use large language models to flag problem... by mensetmanusman on Friday August 16, 2024 @04:27PM (#41270063)
Cancer is anomalous cell behavior.