News for hackers. Stuff that matters.
Re:The Coming of Local LLMs by rcme on Tuesday April 11, 2023 @12:30PM (#35527479)
LLMs as general purpose classifiers is a really big deal, especially because you can give them fuzzy instructions. I know people are worried about LLMs and spam, but I think LLMs may provide an opportunity to elevate online discourse by being more efficient at filtering out spam and low quality commentary.
Re:The Coming of Local LLMs by gamegoblin on Tuesday April 11, 2023 @12:41PM (#35527634)
I already have a custom browser plugin that calls out to GPT (gpt-3.5-turbo is cheap and good enough for this) to classify and filter out low-effort, overly negative, or intellectually dishonest HN comments. It significantly improves the experience on this site.

Bonus points: I had never written a browser plugin, but GPT4 helped me do it in under half an hour.

Re:The Coming of Local LLMs by darkgreene on Tuesday April 11, 2023 @12:47PM (#35527706)
Do you have a repo you can share? I'd be very interested in running my own local copy
Re:The Coming of Local LLMs by tough on Tuesday April 11, 2023 @05:55PM (#35532123)
+1
Re:The Coming of Local LLMs by artificial on Tuesday April 11, 2023 @02:21PM (#35529070)
This sounds great, would love to see some more about this endeavor.
Re:The Coming of Local LLMs by metalliqaz on Tuesday April 11, 2023 @02:24PM (#35529114)
How do you define the cost function for 'low effort'?
Re:The Coming of Local LLMs by gamegoblin on Tuesday April 11, 2023 @02:36PM (#35529333)
I give GPT some examples of what I consider low-effort comments and non-low-effort comments, and then I just ask it to do classification. It's quite good at few-shot classification of fuzzy stuff like this.
Re:The Coming of Local LLMs by jpe90 on Tuesday April 11, 2023 @03:37PM (#35530317)
This sounds super cool, I’m very curious how you implemented it.

I’m only vaguely familiar with the API.. if I had to guess I would say you send system instruction that its job is to filter unwanted content. And you mentioned you give few examples of unwanted content, so maybe those are additional messages along with an instruction like “filter the following html:”

For every web request you want to filter, you would re-send all of those messages followed by the page HTML as the final message. Is that close?

Re:The Coming of Local LLMs by gamegoblin on Tuesday April 11, 2023 @04:32PM (#35531129)
Close, but it's a bit more specialized to just work on HN:

- Examples of unwanted content

- Then I give a large numbered list of comments and ask which numbers should be filtered

- The plugin then just deletes those comment nodes from the DOM. If HN ever updates their HTML I will have to tweak this code.

The reason to send a large list of comments is just to save on costs. It's cheaper to do it this way than one comment at a time.

So the main difference from what you've proposed is GPT never sees the HTML. My code enumerates the comments in the HTML and splices them in to the prompt in a nice numbered list, then does the reverse translation from list number to DOM element in the other direction.

Re:The Coming of Local LLMs by jpe90 on Tuesday April 11, 2023 @06:06PM (#35532241)
Oh that’s clever! Thanks for answering!
Re:The Coming of Local LLMs by xk_id on Tuesday April 11, 2023 @03:42PM (#35530413)
I don’t know how anyone can trust to run code on their machine that they don’t understand and hasn’t been reviewed by a third party.
Re:The Coming of Local LLMs by gamegoblin on Tuesday April 11, 2023 @04:33PM (#35531163)
I understand the code now, I just didn't know how to write a browser plugin and having GPT4 walk me through it was 10x faster than reading a tutorial.
Re:The Coming of Local LLMs by sliken on Tuesday April 11, 2023 @04:23PM (#35531008)
How many line of codes are in the kernel, drivers, and apps on your laptop/desktop?

How sure are you that all that code has been reviewed by a 3rd party? How many CVEs a year impact your laptop/desktop?

Do you have any reason to think that increased productivity with LLM assistance will result in lower quality code? Personally I find LLM assistance increases productivity, decreases the penalty of using a more difficult language like rust, and makes it more palatable to spend more (LLM assisted) time writing tests.

Re:The Coming of Local LLMs by xk_id on Tuesday April 11, 2023 @08:31PM (#35533592)
I meant reviewed by a (competent) human; not, audited by a security team/whatever. Generative computer code is in a whole new category of risk, because it can hallucinate a `rm -rf *` that you might overlook if you don't understand the code yourself.
Re:The Coming of Local LLMs by sliken on Tuesday April 11, 2023 @08:47PM (#35533710)
Well at least for now, any non-trivial code is going to be written by a human+AI, likely one function at a time. It's pretty rare for the AI to give me exactly what I want, often I have to tweak my prompts multiple times, and even just give up and write a function based on the useful bits from previous queries.

I just don't see it chatbot assisted programming any worse than what we have today.

Re:The Coming of Local LLMs by rcme on Tuesday April 11, 2023 @03:47PM (#35530479)
Isn't that pretty much all code?
Re:The Coming of Local LLMs by ChatGTP on Tuesday April 11, 2023 @05:07PM (#35531576)
Some negative, even overly negative comments contain good ideas though so maybe it’s not a great idea?
Re:The Coming of Local LLMs by heyoni on Wednesday April 12, 2023 @12:32AM (#35535032)
Your comment has been detected to contain oppositional tones and has thus been filtered out.

You are hereby removed from the discourse.

/s

Re:The Coming of Local LLMs by ChatGTP on Wednesday April 12, 2023 @08:06AM (#35538328)
Love it!
Re:The Coming of Local LLMs by 4dayworkweek4u on Thursday April 13, 2023 @08:27PM (#35564045)
It's very inclusive to filter out all "negative" people.
Re:The Coming of Local LLMs by duckmysick on Wednesday April 12, 2023 @03:03AM (#35535959)
That's ok. If it's a good idea, somebody else will talk about in a positive way.

Also, unless you are reading every comment in every thread, you are going to miss a few interesting ideas anyway. That's ok too.

Re:The Coming of Local LLMs by ChatGTP on Wednesday April 12, 2023 @08:37PM (#35549242)
Stay calm and keep summarising!
Re:The Coming of Local LLMs by 4dayworkweek4u on Thursday April 13, 2023 @08:25PM (#35564034)
You are handsome, clever and very virtuous... You are in a positivity filter bubble -- that is super positive. (What about DEI though? Are you being inclusive? Filtering is exclusive).

You are so good.. because you don't acknowledge negativity. Best way to solve problems is to ignore them. Let's all be super loving and positive! Yay.

Re:The Coming of Local LLMs by muyuu on Tuesday April 11, 2023 @12:53PM (#35527808)
spam and especially phishing will become much, much better

it will be hard to trust anything at all

Re:The Coming of Local LLMs by yacine_ on Tuesday April 11, 2023 @01:01PM (#35527921)
very simple, just don't read the email
Re:The Coming of Local LLMs by akiselev on Tuesday April 11, 2023 @01:44PM (#35528519)
That’s great. My coworkers are already a step ahead on security!
Re:The Coming of Local LLMs by muyuu on Tuesday April 11, 2023 @01:50PM (#35528611)
I'm half way there, I already have no mates and I ignore most of the email from work.
Re:The Coming of Local LLMs by SV_BubbleTime on Tuesday April 11, 2023 @03:23PM (#35530092)
Which will force drive signed communications. So... win.
Re:The Coming of Local LLMs by muyuu on Tuesday April 11, 2023 @03:46PM (#35530460)
Yea in a sense it will be a win, but it will up the stakes. A lot of people with get scammed. A lot more than right now.
Re:The Coming of Local LLMs by SV_BubbleTime on Tuesday April 11, 2023 @07:26PM (#35533118)
Disagree. Adding signed communications as the default to replace email's completely nonsense zero encryption model is not upping the stakes. Email is the most successful federated software in the universe, but it's fucking time to get some encryption and authn.
Re:The Coming of Local LLMs by muyuu on Tuesday April 11, 2023 @07:39PM (#35533216)
you are having a good laugh if you think the average granny is going to do that anytime soon
Re:The Coming of Local LLMs by MichaelZuo on Tuesday April 11, 2023 @09:17PM (#35533868)
Why would they? If actually implemented it will be a few checkboxes behind a prompt next time they login.
Re:The Coming of Local LLMs by SV_BubbleTime on Tuesday April 11, 2023 @09:48PM (#35534071)
Yea, lol, what was I thinking!? It’s not like my granny uses TLS ECDH SHA1 right!? Oh, wait, everyday she does? The tools adapt; email has never had to.
Re:The Coming of Local LLMs by muyuu on Wednesday April 12, 2023 @06:21AM (#35537432)
the tools will only adapt when a large number of people have got burnt

as always happens

Re:The Coming of Local LLMs by nathan_compton on Tuesday April 11, 2023 @03:00PM (#35529762)
Like almost all machine learning stuff, I expect these things to only be useful in places where it doesn't really matter if the results are correct. When you apply a classifier in real life its critically important to understand its statistical characteristics which is typically done via model characterization, which involves cross validation or boot strapping or whatever. I think the idea that you can just zero-shot or few-shot deploy these things as classifiers and forget about it is incredibly naive unless (as I said earlier) the results just don't really matter.

I've already used LLMs in my work as a data scientist but it requires a ton of work to just make the results tractable (and I have been using GPT4, which behaves pretty well). These smaller language models ain't so regular. Ok, like consider a basic thing you want to do with a classifier: understand its behavior on a held out data set. Since no one knows what is really in the training data (since its so large), its quite hard to understand what the model can generalize about and what it has just accidentally memorized. CF reports that GPT4 doesn't perform nearly as well on even simple programming exercises that are chosen in such a way as to be sure they weren't in the training data.

There is enormous potential for statistical fuck ups here. Prompt engineering, for instance, is an easy place for over-fitting to happen as a prompt is fine tuned on data the prompt engineer has and thus fails to generalize to new data.

I do think there is a lot of value here, but I'm also sure that sloppy use of large language models is going to cause a bunch of trouble in the short to medium term, generate a lot of garbage, pollute a lot of databases, etc, while we figure all this stuff out.

Re:The Coming of Local LLMs by rcme on Tuesday April 11, 2023 @03:24PM (#35530102)
Nothing about a zero or few-shot classifier precludes you from validating on a data set.
Re:The Coming of Local LLMs by nathan_compton on Tuesday April 11, 2023 @03:35PM (#35530278)
You can certainly hold out a validation set while you are writing your prompt, but you can't know whether the model is over fitting for your data set since you don't know what data was in the training set.
Re:The Coming of Local LLMs by rcme on Tuesday April 11, 2023 @03:46PM (#35530466)
You'll probably find out pretty quick in production :)

But I posit that most text classification tasks don't have such strict accuracy requirements. For one, no text classifier is 100% accurate. For instance, I have genuine mail in my spam folder frequently. I see spam on social networks, etc. I struggle to think of cases that aren't at least somewhat tolerant to some amount of incorrect classification.

Re:The Coming of Local LLMs by jprete on Tuesday April 11, 2023 @08:57PM (#35533761)
I think most users are not going to do this; they have no reason to even know such a thing might be done, let alone that it’s useful.