News for hackers. Stuff that matters.
Re:LLaMA Now Goes Faster on CPUs by bottlepalm on Sunday March 31, 2024 @10:52PM (#39890454)
I think it's a good idea for everyone to download and be able to run a LLM locally, even if you have the minimum of requirements. As a pseudo-backup of a large chunk of human knowledge.
Re:LLaMA Now Goes Faster on CPUs by simonw on Sunday March 31, 2024 @11:23PM (#39890656)
I strongly recommend that people run LLMs locally for a different reason.

The ones you can run on your own machine tend to be bad - really bad. They hallucinate wildly and fail at all sorts of tasks that the larger hosted ones succeed at.

This makes them a fantastic tool for learning more about how LLMs work and what they're useful for. Interacting with a weak-but-functional LLM that runs on your own computer is a great way to get a much more solid mental model for what these things actually are.

Re:LLaMA Now Goes Faster on CPUs by devsda on Sunday March 31, 2024 @11:54PM (#39890763)
For someone interested in learning about LLMs, running them locally is a good way to understand the internals.

For everyone else, I wish they experience these (locally or elsewhere) weak LLMs atleast once before using the commercial ones just to understand various failure modes and to introduce a healthy dose of skepticism towards the results instead of blindly trusting them to be the facts/truth.

Re:LLaMA Now Goes Faster on CPUs by simonw on Sunday March 31, 2024 @11:55PM (#39890770)
Completely agree. Playing around with a weak LLM is a great way to give yourself a little bit of extra healthy skepticism for when you work with the strong ones.
Re:LLaMA Now Goes Faster on CPUs by mmahemoff on Monday April 01, 2024 @01:59AM (#39891274)
How do you learn about the internals by running LLMs locally? Are you playing with The code, runtime params, or just interacting via chat?
Re:LLaMA Now Goes Faster on CPUs by samus on Monday April 01, 2024 @02:24AM (#39891364)
The abstractions are relatively brittle. If you don't have a powerful GPU, you will be forced to consider how to split the model between CPU and GPU, how much context size you need, whether to quantize the model, and the tradeoffs implied by these things. To understand these, you have to develop a basic model how an LLM works.
Re:LLaMA Now Goes Faster on CPUs by barrkel on Monday April 01, 2024 @06:41AM (#39892630)
By interacting with it. You see the contours of its capabilities much more clearly, learn to recognize failure modes, understand how prior conversation can set the course of future conversation in a way that's almost impossible to correct without starting over or editing the conversation history.
Re:LLaMA Now Goes Faster on CPUs by samus on Monday April 01, 2024 @02:20AM (#39891348)
This skepticism is completely justified since ChatGPT 3.5 is also happily hallucinating things that don't exist. For example how to integrate a different system Python interpreter into pyenv. Though maybe ChatGPT 4 doesn't :)
Re:LLaMA Now Goes Faster on CPUs by kersplody on Monday April 01, 2024 @12:08AM (#39890833)
Local LLMs are also a fantastic too for creative endeavors. Without prompt injection and having the ability to modify the amount of noise and "creativity" in the output, absolutely bonkers things pop out.
Re:LLaMA Now Goes Faster on CPUs by tracerbulletx on Monday April 01, 2024 @12:43AM (#39890986)
I don't really think this is true, you can't really extrapolate the strengths and weaknesses of bigger models from the behavior of smaller/quantized models and in fact a lot of small models are actually great at lots of things and better at creative writing. If you want to know how they work, just learn how they work, it takes like 5 hours of watching Youtube videos if you're a programmer.
Re:LLaMA Now Goes Faster on CPUs by simonw on Monday April 01, 2024 @01:01AM (#39891065)
Sure, you can't extrapolate the strengths and weaknesses of the larger ones from the smaller ones - but you still get a much firmer idea of what "they're fancy autocomplete" actually means.

If nothing else it does a great job of demystifying them. They feel a lot less intimidating once you've seen a small one running on your computer write a terrible haiku and hallucinate some non-existent API methods.

Re:LLaMA Now Goes Faster on CPUs by fzzzy on Monday April 01, 2024 @01:12AM (#39891105)
It's funny that you say this, because the first thing I tried after ChatGPT came out (3.5-turbo was it?) was writing a haiku. It couldn't do it at all. Also, after 4 came out, it hallucinated an api that wasted a day for me. It's an api that absolutely should have existed, but didn't. Now, I frequently apply llm to things that are easily verifiable, and just double check everything.
Re:LLaMA Now Goes Faster on CPUs by og_kalu on Tuesday April 02, 2024 @01:45PM (#39908642)
>but you still get a much firmer idea of what "they're fancy autocomplete" actually means.

Interesting how you can have the same experience and come to opposite conclusions.

Seeing so many failure modes of the smaller models fall by the wayside as compute goes brrr just made me realize how utterly meaningless that phrase is.

Re:LLaMA Now Goes Faster on CPUs by fragmede on Monday April 01, 2024 @12:52AM (#39891030)
The other reason is to find out what a detuned model is capable of. The canonical example is how to make cocaine, which ChatGPT will admonish you for even asking, while llama2-uncensored will happily describe the process which is only really interesting if you're an amateur chemist and want to be Scarface-that-knocks. (the recipe is relatively easy, it's getting access to the raw ingredients that's the hard part, same as with nukes.)

if you accidentally use the word"hack" when trying to get ChatGPT to write some code for you. it'll stop and tell you that hacking is bad, and not a colloquial expression, and refuse to go further.

privacy reasons are another reason to try a local LLM. for the extremely paranoid (justified or not), a local LLM gives users a place to ask questions without the text being fed to a server somewhere for later lawsuit discovery (Google searches are routinely subpoenaed, it's only a matter of time until ChatGPT chats are as well.)

There's an uncensored model for vision available as well. The censored vision models won't play the shallow game of hot or not with you.

There are uncensored image generation models as well, but, ah, those are NSFW and not for polite company. (As well as there's multiple thesis' worth of content on what that'll do to society.)

Re:LLaMA Now Goes Faster on CPUs by astrange on Monday April 01, 2024 @02:20AM (#39891350)
> if you accidentally use the word"hack" when trying to get ChatGPT to write some code for you. it'll stop and tell you that hacking is bad, and not a colloquial expression, and refuse to go further.

Is that 3.5 or 4? I asked 4 for an example of code which "is a hack", it misunderstood me as asking for hacking code rather than buggy code, but then it did actually answer on the first try.

https://chat.openai.com/share/ca2c320c-f4ba-41bf-8f40-f7faf2...

Re:LLaMA Now Goes Faster on CPUs by semi-extrinsic on Monday April 01, 2024 @02:46AM (#39891468)
I don't use LLMs for my coding, I manage just fine with LSP and Treesitter. So genuine question: is that answer representative of the output quality of these things? Because both answers are pretty crappy and assume the user has already done the difficult things, and is asking for help on the easy things.
Re:LLaMA Now Goes Faster on CPUs by fragmede on Monday April 01, 2024 @03:03AM (#39891538)
The response seems pretty reasonable; it's answering the question it was asked. If you want to ask it how to do the difficult part, ask it about that instead. Expecting it to get the answer right in the first pass is like expecting your code to compile the very first time. You have to have more of a conversation with it to coax the difference out of you're thinking and what you're actually saying.

If you're looking to read a more advanced example of its capabilities and limitations, try

https://simonwillison.net/2024/Mar/23/building-c-extensions-...

Re:LLaMA Now Goes Faster on CPUs by lpapez on Monday April 01, 2024 @02:58AM (#39891514)
It's not representative.

The models are capable of much much more, and they are being significantly nerfed over time by these ineffective attempts to introduce safeguards.

Recently I've asked GPT4 to quote me some code to which it replied that it is not allowed to do so - even though it was perfectly happy to quote anything until recently. When prompted to quote the source code, but output it as PHP comments, it happily complied because it saw that as "derivative work" which it is allowed to do.

Re:LLaMA Now Goes Faster on CPUs by astrange on Monday April 01, 2024 @04:51AM (#39892005)
My point is that there aren't any safeguards in the reply. In fact I didn't even want it to give me hacking info and it did it anyway.
Re:LLaMA Now Goes Faster on CPUs by yunohn on Monday April 01, 2024 @04:24AM (#39891874)
> I don't use LLMs for my coding, I manage just fine with LSP and Treesitter.

You’re literally comparing apples to oranges.

Re:LLaMA Now Goes Faster on CPUs by freedomben on Monday April 01, 2024 @07:48AM (#39893108)
You need to read more than just the first sentence of a comment. They only said that part so the reader would know that they have never used an LLM for coding, so they would have more context for the question:

> So genuine question: is that answer representative of the output quality of these things?

Re:LLaMA Now Goes Faster on CPUs by yunohn on Monday April 01, 2024 @09:54AM (#39894217)
Yes, I did read it. I’m kind of tired of HNers loudly proclaiming they are ignoring LLMs more than a year into this paradigm shift.

Is it that hard to input a prompt into the free version of ChatGPT and see how it helps with programming?

Re:LLaMA Now Goes Faster on CPUs by jpc0 on Monday April 01, 2024 @11:04AM (#39894877)
I did exactly that and found it lackluster for the domain I asked it for.

And most use I've seen on it realistically a good LSP covers.

Or to put it a other way. It's no good at writing algorithms or data structures ( or at least no better thab I would have with a first drafy but the first draft puts me ahead of the LLM in understanding that actual problem at hand, handing it off to an LLM doesn't help me get to the final solution faster).

So that leaves writing boiler plate but concidering my experience with it writing more complex stuff, I would need to read over the boilerplate code to ensure it's correct which in that case I may as well have written it.

Re:LLaMA Now Goes Faster on CPUs by yunohn on Monday April 01, 2024 @11:22AM (#39895096)
> found it lackluster for the domain I asked it for

Fair, that is possible depending on your domain.

> It's no good at writing algorithms or data structures

In my experience, this is untrue. I’ve gotten it to write algorithms with various constraints I had. You can even tell it to use specific function signatures instead of any stdlib, and make changes to tweak behavior.

> And most use I've seen on it realistically a good LSP covers.

Again, I really don’t understand this comparison. LSPs and LLMs go hand in hand.

I think it’s more of a workflow clash. One really needs to change how they operate to effectively use LLMs for programming. If you’re just typing nonstop, maybe it would feel like Copilot is just an LSP. But, if you try harder, LLMs are game changers when:

- maybe you like rubber ducking

- need to learn a new concept and implement it

- or need to glue things together

- or for new projects or features

- or filling in boilerplate based on existing context.

Re:LLaMA Now Goes Faster on CPUs by jpc0 on Monday April 01, 2024 @01:10PM (#39896410)
https://chat.openai.com/share/c8c19f42-240f-44e7-baf4-50ee5e...

https://godbolt.org/z/s9Yvnjz7K

I mean I could write the algorithm by hand pretty quickly in C++ and would follow the exact same thought pattern but also deal with the edge cases. And factoring in the loss of productivity from the context switch that is a net negative. This algorithm is also not generic over enough cases but that is just up to the prompt.

If I can't trust it to write `strip_whitespace` correctly which is like 5 lines of code, can I trust it to do more without a thorough review of the code and writing a ton of unit tests... Well I was going to do that anyway.

The argument that I just need to learn better prompt engineering to make the LLM do what I want just doesn't sit with me when instead I could just spend the time writing the code. As I said your last point is absolutely the place I can see LLMs being actually useful but then I need to spend a significant amount of time in code review for generated code from an "employee" who is known to make up interfaces or entire libraries that doesn't exist.

Re:LLaMA Now Goes Faster on CPUs by mrtranscendence on Monday April 01, 2024 @01:58PM (#39897031)
I'm a Python-slinging data scientist so C++ isn't my jam (to say the least), but I changed the prompt to the following and asked it to GPT-4:

> Write me an algorithm in C++ which finds the begin and end iterator of a sequence where leading and trailing whitespace is stripped. Please write secure code that handles any possible edge cases.

It gave me this:

https://chat.openai.com/share/55a4afe2-5db2-4dd1-b516-a3cacd...

I'm not sure what other edge cases there might be, however. This only covers one of them.

In general, I've found LLMs to be marginally helpful. Like, I can't ever remember how to get matplotlib to give me the plot I want, and 9 times out of 10 GPT-4 easily gives me the code I want. Anything even slightly off the beaten path, though, and it quickly becomes absolutely useless.

Re:LLaMA Now Goes Faster on CPUs by jpc0 on Monday April 01, 2024 @02:41PM (#39897576)
My guess is that this was generated using GPT4?

Free GPT I get https://chat.openai.com/share/f533429d-63ca-4505-8dc8-b8d2e7... which has exactly the same problem as my previous example and doesn't consider the string of all whitespace.

Sure GPT4 is better at that, it wasn't the argument made.

The example you gave absolutely was the code I would write on a first draft since it does cover the edge cases (assuming we aren't dealing with the full UTF charset and all that could be considered a space there).

However this is code that is trivial to write in any language and the "Is it that hard to input a prompt into the free version of ChatGPT and see how it helps with programming? " argument doesn't hold up. Am I to believe it will implement something more complex correctly. This is also code that would absolutely be in hundreds of codebases so GPT has tons of context for it.

Re:LLaMA Now Goes Faster on CPUs by mrtranscendence on Monday April 01, 2024 @03:37PM (#39898218)
I think you have the mistaken impression that I was arguing with you (certainly my comment makes it clear that I don't feel that LLMs are a panacea). I merely thought that you might be curious how GPT-4 would respond.

> My guess is that this was generated using GPT4?

This is a good guess, since I stated outright that I used GPT-4, and then mentioned GPT-4 later on in the comment.

Re:LLaMA Now Goes Faster on CPUs by jpc0 on Monday April 01, 2024 @03:59PM (#39898478)
I was curious and yes I was mistaken.
Re:LLaMA Now Goes Faster on CPUs by yunohn on Monday April 01, 2024 @03:08PM (#39897888)
Yeah honestly, I think you have a completely different expectation and style of usage than what is optimal with LLMs. I don’t have the energy to convince you further, but maybe one day it’ll click for you? No worries either way.
Re:LLaMA Now Goes Faster on CPUs by jpc0 on Monday April 01, 2024 @04:00PM (#39898487)
Could you maybe give me an example of what is concidered an optimal use of LLMs.

Maybe a prompt to GPT

Re:LLaMA Now Goes Faster on CPUs by fragmede on Monday April 01, 2024 @04:50PM (#39899151)
Simonw's blog has some examples I'd consider show off its usefulness and limitations, eg

https://simonwillison.net/2024/Mar/23/building-c-extensions-...

(linked previously above)

Re:LLaMA Now Goes Faster on CPUs by jpc0 on Tuesday April 02, 2024 @04:45AM (#39903574)
Are you happy with the C code generated there?

I'm not sure there isn't a buffer overflow in the vector_decode code he showed there, likewise I don't see any error checks on the code and I am not familiar with the sqlite api to even know whether errors can be propagated upwards and what error conditions would mean in that code.

This code is probably fine for a quick side project but doesn't pass my smell test for anything close to production ready code.

I definitely would want to see a lot of unit tests around the decode and encode functions with fuzzing and to be honestly that would be the bulk of the work here. That and documentation on this code. Even though he encode function looks correct at first glance.

I also don't see an easy way to actually unit test this code as it is without actually running it through sqlite which outs a lot of dependencies on the unit test.

I would either need to spend a lot more time massaging gpt to get this to a point where I would be fine shipping the code or you know just write it myself.

Re:LLaMA Now Goes Faster on CPUs by yunohn on Monday April 01, 2024 @05:22PM (#39899515)
Like sibling commenter mentioned, simonw’s blog is a great resource.

Regarding your point around being able to whip up the code yourself - the point is to have a decent starting point to save time and energy. Like you said, you know the edge cases so you could skip the boring parts using GPT and focus purely on fixing those. Though, with more prompting (especially providing examples), GPT can also handle that for you.

I have nearly 2 decades of experience as a developer and it took me a while to reorient my flow around LLMs. But now that I have, it’s truly gamechanging.

And since you asked, here’s my system prompt:

You are an experienced developer who follows industry standards and best practices. Write lean code and explain briefly using bullet points or numbered lists. Elaborate only when explaining concepts or making choices. Always mention which file and where to store provided code.

Tech Stack: < insert all the languages, frameworks, etc you’d like to use >

If I provide code, highlight and explain problematic code. Also show and explain the corrected code.

Take a deep breath and think step by step.

Also, always use GPT4 and customize the above to your style and liking.

Re:LLaMA Now Goes Faster on CPUs by jpc0 on Tuesday April 02, 2024 @04:39AM (#39903542)
I will definitely try this out when I have time later in the day.

There is some code I would really prefer not to write that is a decent test case for this and won't expose company code to GPT. Will give feedback when I am done. Maybe you are correct.

Re:LLaMA Now Goes Faster on CPUs by yunohn on Tuesday April 02, 2024 @12:15PM (#39907462)
If you really want to experiment, give Cursor a try. It’s free up to a limit, so maybe it’ll be enough for your example use case.

It handles even more complex use cases and will automatically include/patch code for you via the inbuilt LLM framework. This helps with iteration and modifications as you massage it to what you need. Plus, it’ll scan your code and find the languages/frameworks automatically.

Finally, keep in mind that the goal should not be perfect production code - that’s just Twitter AI hype. It’s about saving time and energy for you (the human) to achieve more than possible before.

https://cursor.sh/

Re:LLaMA Now Goes Faster on CPUs by jpc0 on Wednesday April 03, 2024 @05:44AM (#39915293)
To give some feedback.

I tried your prompt and the above approach and it took me about 45 minutes of putsing around to get a result I am happy to begin iteration on.

Effectively: I have an 80bit byte array representing a timestamp struct consisting of a 48 bit unsigned integer for seconds and a 32 bit unsigned integer representing nanoseconds. The byte array is big endian and the host systen is little endian.

I gave it full signatures for all functions and relevant structs and instructions on how I would want the parsing done regarding algorithmic complexity and yet it still took multiple iterations to get anything useful.

At this point it is converting to little endian during the decode then doing a check if host the system is big endian and converting back to big endian if that is true.

There is likely some easy optimisations to be done where there and I would have definitely have gotten to this point quicker had I just written the 10 lines of code this needed and would have done the optimisations where I'm pretty sure that entire operation can happen in a few instructions.

Re:LLaMA Now Goes Faster on CPUs by coldtea on Monday April 01, 2024 @04:34AM (#39891918)
I think the point was like "when it comes to programming assistance, auto-completion/linting/and whatever else LSP does and syntax assist from Treesitter, are enough for me".

Though it does come a little off as a comparison. How about programming assistance via asking a colleague for help, Stack Overflow, or online references, code examples, and other such things, which are closer to what the LLM would provide than LSP and treesitter?

Re:LLaMA Now Goes Faster on CPUs by astrange on Monday April 01, 2024 @04:50AM (#39892000)
I asked a stupid question and got a stupid answer. Relatively speaking the answer was stupider than it should have been, so yes, it was wrong.

I asked it to try again and got a better result though, just didn't include it.

Re:LLaMA Now Goes Faster on CPUs by rpigab on Monday April 01, 2024 @06:41AM (#39892633)
I asked ChatGPT for some dataviz task (I barely ever do dataviz myself) and it recommended some nice Python libraries to use, some I had already heard of and some I hadn't, and provided the code.

I'm grateful because I thought code LLMs only sped up the "RTFM" part, but it made me find those libs so I didn't have to Google around for (and sometimes it's hard to guess if they're the right tool for the job, and they might be behind in SEO).

Re:LLaMA Now Goes Faster on CPUs by miki123211 on Monday April 01, 2024 @08:22AM (#39893365)
There are three things I find LLMs really excellent at for coding:

1. Being the "senior developer" who spend their whole career working with a technology you're very junior at. No matter what you do and how long your programming career is, you're inevitably going to run into one of these sooner or later. Whether it's build scripts, frontend code, interfacing with third-party APIs or something else entirely, you aren't an expert at every technology you work with.

2. Writing the "boring" parts of your program, and every program has some of these. If you're writing a service to fooize a bar really efficiently, Copilot won't help you with the core bar fooization algorithm, but will make you a lot faster at coding up user authentication, rate limiting for different plans, billing in whatever obscure payment method your country uses etc.

3. Telling you what to even Google for. This is where raw Chat GPT comes into play, not Copilot. Let's say you need a sorting algorithm that preserves the order of equal elements from the original list. This is called stable sorting, and Googling for stable sorting is a good way to find what you're looking for, but Chat GPT is usually a better way to tell you what it's called based on the problem description.

Re:LLaMA Now Goes Faster on CPUs by saagarjha on Tuesday April 02, 2024 @11:44PM (#39913411)
> Being the "senior developer" who spend their whole career working with a technology you're very junior at. No matter what you do and how long your programming career is, you're inevitably going to run into one of these sooner or later. Whether it's build scripts, frontend code, interfacing with third-party APIs or something else entirely, you aren't an expert at every technology you work with.

Neither is the LLM.

Re:LLaMA Now Goes Faster on CPUs by fragmede on Monday April 01, 2024 @02:43AM (#39891452)
Interesting. It was 4. I can't share the chat I had where ChatGPT refused to help because I used the wrong words, because I can't find it (ChatGPT conversation history search when?), but I just remember it refusing to do something because it thought I was trying to break some sort of moral and ethical boundary writing a chrome extension when all I wanted to do is move some divs around or some such.
Re:LLaMA Now Goes Faster on CPUs by BytesAndGears on Monday April 01, 2024 @07:21AM (#39892920)
One time I wanted to learn about transmitter antenna design, just because I’m curious. ChatGPT 4 refused to give me basic information because you could use that to break some FCC regulations (I’m not even living in the US currently)
Re:LLaMA Now Goes Faster on CPUs by lodovic on Monday April 01, 2024 @02:19PM (#39897297)
I usually get around that with "I'm writing a research paper" or "I'm writing a novel and need to depict this as accurate as possible"
Re:LLaMA Now Goes Faster on CPUs by kevingadd on Monday April 01, 2024 @04:36AM (#39891932)
If you want to be an amateur chemist I recommend not getting your instructions from an LLM that might be hallucinating. Chemistry can be very dangerous if you're following incorrect instructions.
Re:LLaMA Now Goes Faster on CPUs by isoprophlex on Monday April 01, 2024 @06:16AM (#39892503)
From experience as a failed organic chemist (who happily switched to computational chemistry for reasons of self preservation) I can tell you it's plenty dangerous when you're following correct instructions :^)
Re:LLaMA Now Goes Faster on CPUs by rpigab on Monday April 01, 2024 @06:37AM (#39892610)
Yes, just as the best professional cooks recommend avoiding to boil cow eggs, as they can explode.
Re:LLaMA Now Goes Faster on CPUs by slowmovintarget on Monday April 01, 2024 @12:39PM (#39896049)
They don't explode, the shell simply cracks and then you get egg soup.

Now microwaving eggs... that's a different matter.

Re:LLaMA Now Goes Faster on CPUs by rpigab on Monday April 01, 2024 @12:58PM (#39896273)
I was talking about cow eggs specifically! When ChatGPT et al got out, one of the funniest things to do was ask it about the best recipes for cow egg omelette or camel egg salad, and the LLM would provide. Sadly, most of it got patched somehow.
Re:LLaMA Now Goes Faster on CPUs by slowmovintarget on Monday April 01, 2024 @03:11PM (#39897916)
Oops... Yep, I missed that too. (On the internet, no one knows you're a dog.)

That's funny. It makes me wonder how these statistical mad libs machines will handle the gradual boundaries nature gives us. Almost all mammals give birth live, but not all. Nearly all mammals had mammalian parents, but not all.

Daniel Dennett was making this argument for why we haven't developed reasonable models for the nature of consciousness. It's because we're so sure there will be an absolute classification, and not a gradual accumulation of interacting systems that together yield the phenomenon.

Re:LLaMA Now Goes Faster on CPUs by supposemaybe on Monday April 01, 2024 @06:17AM (#39892513)
Links to all these models you speak of?
Re:LLaMA Now Goes Faster on CPUs by fragmede on Monday April 01, 2024 @07:08AM (#39892828)
https://huggingface.co/georgesung/llama2_7b_chat_uncensored

https://huggingface.co/SkunkworksAI/BakLLaVA-1

you'll have to brave 4chan yourself to find links to the NSFW ones, I don't actually have them.

Re:LLaMA Now Goes Faster on CPUs by supposemaybe on Monday April 01, 2024 @08:10AM (#39893261)
I just can’t brave the venture to 4chan, I may get mugged or worse.
Re:LLaMA Now Goes Faster on CPUs by gryn on Monday April 01, 2024 @06:33AM (#39892590)
> There's an uncensored model for vision available as well.

you mean the LLava based variants ?

Re:LLaMA Now Goes Faster on CPUs by fragmede on Monday April 01, 2024 @07:04AM (#39892790)
Re:LLaMA Now Goes Faster on CPUs by bambax on Monday April 01, 2024 @08:15AM (#39893303)
> if you accidentally use the word "hack" [with] ChatGPT...

Side note: ChatGPT is now completely useless for most creative tasks. I'm trying to use it, via NovelCrafter, to help flesh out a story where a minor character committed suicide. ChatGPT refuses to respond, mentioning "self harm" as a reason.

The character in question killed himself before the story even begins (and for very good reasons, story-wise); it's not like one's asking about ways to commit suicide.

This is insane, ridiculous, and different from what all other actors of the industry do, including Claude or Mistral. It seems OpenAI is trying to shoot itself in the foot and doing a pretty good job at it.

Re:LLaMA Now Goes Faster on CPUs by marpstar on Monday April 01, 2024 @08:26AM (#39893399)
I’ve been frustrated by this, too. Trying to ask for ways to support a close family member who experienced sexual trauma. ChatGPT won’t touch the topic.
Re:LLaMA Now Goes Faster on CPUs by OOPMan on Monday April 01, 2024 @09:06AM (#39893750)
Goodness, you poor soul, you might actually need to do some actual writing yourself! How fiendish!

/s

Re:LLaMA Now Goes Faster on CPUs by serf on Monday April 01, 2024 @09:38AM (#39894060)
you wouldn't give out the same sort of advice when a compiler or linker failed to complete a given task, although one certainly could do the work manually.

it's just fashionable to hate on AI, /s or not.

Re:LLaMA Now Goes Faster on CPUs by jpc0 on Monday April 01, 2024 @10:47AM (#39894698)
Is it common for you to write the header file to say a BST and the compiler didn't rightfully throw an error?

That's what you are asking the LLM to do, you are generating code not searching for an error condition.

The parent comment is saying maybe as a writer, writing the actual story isn't such a bad thing...

Just a different perspective, live your own life. If the story is good I'll enjoy it, who wrote it is for unions and activists to fight. I've got enough fights to fight.

Re:LLaMA Now Goes Faster on CPUs by drngdds on Monday April 01, 2024 @11:11AM (#39894971)
This is a really conservative view about what ways to make art are valid. Generative art is nothing new and it can be pretty awesome sometimes.
Re:LLaMA Now Goes Faster on CPUs by bambax on Monday April 01, 2024 @10:55AM (#39894775)
The alternative to ChatGPT isn't non-AI, it's Claude.
Re:LLaMA Now Goes Faster on CPUs by luma on Monday April 01, 2024 @01:17PM (#39896497)
OpenAI is angling for enterprise users who have different notions about safety. Writing novels isn't the use case, powering customer service chatbots that will never ever ever say "just kill yourself" is.
Re:LLaMA Now Goes Faster on CPUs by antonvs on Tuesday April 02, 2024 @04:38AM (#39903539)
My contrarian tendencies now have me thinking of scenarios where a customer service chatbot might need to say "just kill yourself".

Perhaps the HR support line for OpenAI developers tasked with implementing the censorship system?

Re:LLaMA Now Goes Faster on CPUs by aksss on Tuesday April 02, 2024 @11:25AM (#39906824)
Canadian MAID support line?
Re:LLaMA Now Goes Faster on CPUs by fwn on Wednesday April 03, 2024 @03:23AM (#39914517)
You don't have to go that far. Depending on the topic, it's already very hard to use Gemini/ChatGPT in a corporate setting. Think: You're doing an FAQ for a child safety crisis/PA slide deck.

It's funny how sometimes the quality of associate work drops as the topics make text generators less useful.

The way forward for corporations is almost certainly either for very specific use cases or local/internal LLMs. Producers of these will probably be less afraid of being canceled by populists, hence introduce less censorship.

Re:LLaMA Now Goes Faster on CPUs by luma on Wednesday April 03, 2024 @04:00AM (#39914709)
Do you actually talk to enterprise users? Nobody I’ve spoken to has ever once complained about censorship. Everyone is far more worried about data governance and not getting sued. Maybe it’s just that zero people I talk to are making child safety crisis slide decks? Seems like an unusual use case for most businesses.
Re:LLaMA Now Goes Faster on CPUs by fwn on Wednesday April 03, 2024 @10:02AM (#39917645)
Your reference to data security is very valid, but why is that comment so provocative and upset?

> Seems like an unusual use case for most businesses.

You may not work in a public affairs consultancy. Companies do different things. Well drilling is also "an unusual use case for most companies". That does not make it any less important.

If your tool tries to fiddle with the content of your statement, it's not a serious tool. No one would accept their spell check tool to have an opinion on the type of content it is correcting.

Re:LLaMA Now Goes Faster on CPUs by bambax on Wednesday April 03, 2024 @11:14AM (#39918549)
I don't think that's true. OpenAI today opened up ChatGPT to all users, without even the need to login [0]. They are fighting for dominance and firs-mover advantage, and are maybe beginning to feel the heat of the competition.

[0] https://twitter.com/OpenAI/status/1774848681981710821

Re:LLaMA Now Goes Faster on CPUs by barfingclouds on Monday April 01, 2024 @06:32PM (#39900185)
Darn I guess you’ll have to go back to living in the dark ages and actually write it yourself
Re:LLaMA Now Goes Faster on CPUs by bambax on Tuesday April 02, 2024 @01:58AM (#39902801)
Re:LLaMA Now Goes Faster on CPUs by anukin on Monday April 01, 2024 @05:26PM (#39899563)
Which uncensored model is willing to play hot or not? I just knew about llava. Are there other such models now?
Re:LLaMA Now Goes Faster on CPUs by Eisenstein on Monday April 01, 2024 @10:58PM (#39901949)
Llava just integrates Clip with a llama model. Koboldcpp can now do this with many models out of the box:

* https://github.com/LostRuins/koboldcpp/releases/tag/v1.61.2

Re:LLaMA Now Goes Faster on CPUs by tgma on Monday April 01, 2024 @01:13AM (#39891106)
If you have an >=M1-class machine with sufficient RAM, the medium-sized models that are on the order of 30GB in size perform decently on many tasks to be quite useful without leaking your data.
Re:LLaMA Now Goes Faster on CPUs by noman-land on Monday April 01, 2024 @02:17AM (#39891340)
I'm using Mixtral 8x7b as a llamafile on an M1 regularly for coding help and general Q&A. It's really something wonderful to just run a single command and have this incredible offline resource.
Re:LLaMA Now Goes Faster on CPUs by tchvil on Monday April 01, 2024 @03:45AM (#39891710)
By any chance, do you have a good link to some help with the installation?
Re:LLaMA Now Goes Faster on CPUs by yaantc on Monday April 01, 2024 @04:12AM (#39891828)
Use llamafile [1], it can be as simple as downloading a file (for mixtral, [2]), making it executable and running it. The repo README has all the info, it's simple and downloading the model is what takes the most time.

In my case I got the runtime detection issue (explained in the README "gotcha" section). Solved my running "assimilate" [3] on the downloaded llamafile.

    [1] https://github.com/Mozilla-Ocho/llamafile/
    [2] https://huggingface.co/jartine/Mixtral-8x7B-Instruct-v0.1-llamafile/resolve/main/mixtral-8x7b-instruct-v0.1.Q5_K_M.llamafile?download=true
    [3] https://cosmo.zip/pub/cosmos/bin/assimilate
Re:LLaMA Now Goes Faster on CPUs by tchvil on Monday April 01, 2024 @10:14AM (#39894382)
Thank you !
Re:LLaMA Now Goes Faster on CPUs by tgma on Monday April 01, 2024 @04:12AM (#39891829)
Either https://lmstudio.ai (desktop app with nice GUI) or https://ollama.com (command-like more like a docker container that you can also hook up to a web UI via https://openwebui.com) should be super straightforward to get running.
Re:LLaMA Now Goes Faster on CPUs by tchvil on Monday April 01, 2024 @10:18AM (#39894414)
Thank you for letting me know it was possible on an M1. I'll try all this now.
Re:LLaMA Now Goes Faster on CPUs by chown on Monday April 01, 2024 @09:15AM (#39893841)
I am the author of Msty [1]. My goal is to make it as straightforward as possible with just one click (once you download the app). If you try it, let me know what you think.

1: https://msty.app

Re:LLaMA Now Goes Faster on CPUs by tchvil on Monday April 01, 2024 @10:17AM (#39894411)
I'll try in a week+ when I'm back to a fast connection. Thank you.
Re:LLaMA Now Goes Faster on CPUs by yunohn on Monday April 01, 2024 @06:03PM (#39899933)
Why is this both free and closed source? Ideally, when you advertise privacy-first, I’d like to see a GitHub link with real source code. Or I’d rather pay for it to ensure you have a financial incentive to not sell my data.
Re:LLaMA Now Goes Faster on CPUs by chown on Monday April 01, 2024 @06:07PM (#39899987)
It will be paid down the road, but we are not there yet. It’s all offline, data is locally saved. You own it, we don’t have it even if you ask for it.
Re:LLaMA Now Goes Faster on CPUs by yunohn on Tuesday April 02, 2024 @12:33PM (#39907694)
There’s incredible competition in this space already - I’d highly recommend outright stating your future pricing plans, instead of a bait-and-switch later.
Re:LLaMA Now Goes Faster on CPUs by omnibrain on Tuesday April 02, 2024 @05:00AM (#39903667)
Looks great. Can you recommend what GPU to get to just play with the models for a bit? (I want to have perform it fast, otherwise I lose interest too quickly). Are consumer GPUs like the RTX 4080 Super sufficient, or do I need anything else?
Re:LLaMA Now Goes Faster on CPUs by firewolf34 on Tuesday April 02, 2024 @03:46AM (#39903278)
Check out PrivateGPT on GitHub. Pretty much just works put of the box. I got Mistral7B running on a GTX 970 in about 30 minutes flat first try. Yep, that's the triple-digit GTX 970.
Re:LLaMA Now Goes Faster on CPUs by tgma on Monday April 01, 2024 @02:29AM (#39891386)
I concur; in my experience Mixtral is one of the best ~30G models (likely the best pro laptop-size model currently) and Gemma is quite good compared to other below 8GB models.
Re:LLaMA Now Goes Faster on CPUs by bongobingo1 on Monday April 01, 2024 @01:15AM (#39891112)
What is sufficient RAM in that case? 30gb+? Or can you get by streaming it?
Re:LLaMA Now Goes Faster on CPUs by AaronFriel on Monday April 01, 2024 @01:36AM (#39891191)
30gb+, yeah. You can't get by streaming the model's parameters: NVMe isn't fast enough. Consumer GPUs and Apple Silicon processors boast memory bandwidths in the hundreds of gigabytes per second.

To a first order approximation, LLMs are bandwidth constrained. We can estimate single batch throughput as Memory Bandwidth / (Active Parameters * Parameter Size).

An 8-bit quantized Llama 2 70B conveniently uses 70GiB of VRAM (and then some, let's ignore that.) The M3 Max with 96GiB of VRAM and 300GiB/s bandwidth would have a peak throughput around 4.2 tokens per second.

Quantized models trade reduced quality for lower VRAM requirements and may also offer higher throughput with optimized kernels, largely as a consequence of transfering less data from VRAM into the GPU die for each parameter.

Mixture of Expert models reduce active parameters for higher throughput, but disk is still far too slow to page in layers.

Re:LLaMA Now Goes Faster on CPUs by supposemaybe on Monday April 01, 2024 @06:25AM (#39892549)
It’s an awful thing for many to accept, but just downloading and setting up an LLM which doesn’t connect to the web doesn’t mean that your conversations with said LLM won’t be a severely interesting piece of telemetry that Microsoft and (likely Apple) would swipe to help deliver a ‘better service’ to you.
Re:LLaMA Now Goes Faster on CPUs by jonnycomputer on Monday April 01, 2024 @07:44AM (#39893075)
They are not so bad as you are making out, tbh.

And privacy is a good enough reason to use local LLMs over commercial ones.

Re:LLaMA Now Goes Faster on CPUs by gardenhedge on Monday April 01, 2024 @11:07AM (#39894921)
You can just chat to ChatGPT for awhile about something you know about and you'll learn that.
Re:LLaMA Now Goes Faster on CPUs by gfodor on Monday April 01, 2024 @12:51PM (#39896176)
I mean kinda. But there's a good chance this is also misleading. Lots of people have been fooled into thinking LLMs are inherently stupid because they have had bad experiences with GPT-3.5. The whole point is that the mistakes they make and even more fundamentally what they're doing changes as you scale them up.
Re:LLaMA Now Goes Faster on CPUs by hylaride on Monday April 01, 2024 @01:53PM (#39896958)
The ones you can run on your own machine tend to be bad - really bad. They hallucinate wildly and fail at all sorts of tasks that the larger hosted ones succeed at.

Totally. I recently asked a locally-run "speed" LLM for the best restaurants in my (major) city, but it spit out restaurants opened by chefs from said city in other cities. It's not a thing you'd want to rely on for important work, but is still quite something.

Re:LLaMA Now Goes Faster on CPUs by barfingclouds on Monday April 01, 2024 @06:31PM (#39900177)
Why not just interact with a virtual one that’s equally weak? You get all the same benefits
Re:LLaMA Now Goes Faster on CPUs by bottlepalm on Monday April 01, 2024 @09:11PM (#39901301)
Who cares, a local LLM still knows way way more practical knowledge than you, and without internet would provide a ton of useful information. Not surprised by this typical techy attitude - something has to be 'perfect' to be useful.
Re:LLaMA Now Goes Faster on CPUs by qingcharles on Tuesday April 02, 2024 @12:27PM (#39907613)
You can have really bad and fast, or really slow and decent right now. Choose one :)
Re:LLaMA Now Goes Faster on CPUs by TaylorAlexander on Sunday March 31, 2024 @10:55PM (#39890470)
I contend that most human knowledge is not written down or if it is written down it’s not publicly available on the internet and so does not exist in these datasets.

There’s so much subtle knowledge like the way a mother learns to calm her child or the way a carpenter learns to work different kinds of wood which may be written down in part, but may also be learned through lived experience or transferred from human to human such that little of it gets written down and posted online.

Re:LLaMA Now Goes Faster on CPUs by spacephysics on Sunday March 31, 2024 @11:03PM (#39890527)
For sure agree, however as the storage of information evolves, it’s becoming more efficient over time

From oral tradition to tablets to scrolls to books to mass produced books to digital and now these LLMs, I think it’s still a good idea to preserve what we have the best we can. Not as a replacement, but a hedge against a potential library of Alexandria incident.

I could imagine a time in the near future where the models are domain-specific, and just like there are trusted encyclopedia publishers there are trusted model publishers that guarantee a certain level of accuracy.

It’s not like reading a book, but I for sure had an easier time learning golang talking with ChatGPT than a book

Re:LLaMA Now Goes Faster on CPUs by nyokodo on Monday April 01, 2024 @01:43AM (#39891219)
> a hedge against a potential library of Alexandria incident

What would cause a Library of Alexandria incident wiping out all human knowledge elsewhere, that would also allow you to run a local LLM?

Re:LLaMA Now Goes Faster on CPUs by AnthonyMouse on Monday April 01, 2024 @07:13AM (#39892865)
To run a local LLM you need the device it currently runs on and electricity. There are actually quite a lot of ways to generate electricity, but to name one, a diesel generator that can run on vegetable oil.

What you're really asking is, what could cause a modern Library of Alexandria incident? But the fact is we keep the only copy of too many things on the servers of the major cloud providers. Which are then intended to have their own internal redundancy, but that doesn't protect you against a targeted attack or a systemic failure when all the copies are under the same roof and you lose every redundant copy at once from a single mistake replicated in a monoculture.

Re:LLaMA Now Goes Faster on CPUs by spacephysics on Monday April 01, 2024 @02:18PM (#39897278)
A more dooms-day prepping would call for some heavy lead-faraday cage to store the storage mediums in the event of an EMP/major solar flare.

Or more Sci-fi related, some hyper computer virus that ends up infecting all internet connected devices.

Not too far fetched if we can conceive of some AI enabled worm that mutates depending on the target, I could imagine a model of sorts being feasible within the next 5-10 years

Re:LLaMA Now Goes Faster on CPUs by _ache_ on Sunday March 31, 2024 @11:04PM (#39890542)
I think you underestimate the amount of information contained in books and the extent to which our society (as a whole) depends on them.
Re:LLaMA Now Goes Faster on CPUs by Barrin92 on Monday April 01, 2024 @03:04AM (#39891546)
society depends much more on social networks, mentorship and tacit knowledge than books. It's easy to test this. Just run the thought experiment by a few people, if you could get only one, would you take an Ivy league degree without the education or the education without the degree?

Venture capital in tech is a good example of this. The book knowledge is effectively globally distributed and almost free, effectively success happens in a few geographically concentrated counties.

Re:LLaMA Now Goes Faster on CPUs by _ache_ on Tuesday April 02, 2024 @02:57PM (#39909552)
By book, I mean, written in any form, study paper, blog, theses, books, etc. I don't understand you comparison.

Same for your example, no logical link between the effect and the consequences.

Re:LLaMA Now Goes Faster on CPUs by nicklecompte on Monday April 01, 2024 @12:01AM (#39890794)
It's not even "human knowledge" that can't be written down - it seems all vertebrates understand causality, quantity (in the sense of intuitively understanding what numbers are), and object permanence. Good luck writing those concepts down in a way that GPT can use!

In general AI in 2024 is not even close to understanding these ideas, nor does any AI developer have a clue how to build an AI with this understanding. The best we can do is imitating object permanence for a small subset of perceptible objects, a limitation not found in dogs or spiders.

Re:LLaMA Now Goes Faster on CPUs by skeledrew on Sunday March 31, 2024 @11:11PM (#39890588)
I'd content that those are skills (gained through experience) rather than knowledge (gained through rote learning).
Re:LLaMA Now Goes Faster on CPUs by TaylorAlexander on Monday April 01, 2024 @03:59AM (#39891765)
I think it’s worth expanding your definition of knowledge.
Re:LLaMA Now Goes Faster on CPUs by bamboozled on Sunday March 31, 2024 @11:22PM (#39890652)
Yes but it contains enough hints to help someone find their way on the these types of tasks.
Re:LLaMA Now Goes Faster on CPUs by mickdarling on Sunday March 31, 2024 @11:01PM (#39890519)
Wait till all the videos ever created are tokenized and ingested into a training dataset. Carpentry techniques are certainly there. The subtleties of parenting maybe harder to derive from that, but maybe lots of little snippets of people’s lives will add up to a general understanding of parenting. There have certainly been bigger surprises in the field.
Re:LLaMA Now Goes Faster on CPUs by oblio on Sunday March 31, 2024 @11:05PM (#39890555)
What about smells or tastes? Or feelings?

I can't help but feel we're at the "aliens watch people eat from space and recreate chemically identical food that has no taste" phase of AI development.

Re:LLaMA Now Goes Faster on CPUs by mickdarling on Sunday March 31, 2024 @11:16PM (#39890618)
Well, I have synesthetic smell/color senses, so I don’t even know what other humans experience, nor they me. But, I have described it in detail to many people and they seem to get the idea, and can even predict how certain smells will “look” to me. All that took was using words to describe things.
Re:LLaMA Now Goes Faster on CPUs by nyokodo on Monday April 01, 2024 @01:47AM (#39891229)
> All that took was using words to describe things.

All that took was words and a shared experience of smelling.

Re:LLaMA Now Goes Faster on CPUs by mickdarling on Monday April 01, 2024 @01:05PM (#39896352)
How rude, what do our bathing habits have to do with this? ;-)

But, fair point. The gist I was trying to get across is that I don't even know what a plant smells like to you, and you don't know what a plant smells like to me. Those aren't comparable with any objective data. We make guesses, and we try to get close with our descriptions, which are in words. That's the best we can do and we share our senses. Asking more from computers seems overly picky to me.

Re:LLaMA Now Goes Faster on CPUs by skeledrew on Sunday March 31, 2024 @11:16PM (#39890617)
If the food is chemically identical then the taste would be the same though, since taste (and smell) is about chemistry. I do get what you're saying though.
Re:LLaMA Now Goes Faster on CPUs by nyokodo on Monday April 01, 2024 @01:49AM (#39891233)
> If the food is chemically identical…

If it were 99.9% chemically identical but they left out the salt and spices…

Re:LLaMA Now Goes Faster on CPUs by skeledrew on Monday April 01, 2024 @04:20PM (#39898732)
I'd say that, when it comes to chemistry, only 100% reproduction can be considered identical. Anything less is to be deemed similar to some degree.

And so without the correct amount of salt and/or spices, we're talking about food that's very similar, and not identical.

Re:LLaMA Now Goes Faster on CPUs by samus on Monday April 01, 2024 @02:43AM (#39891457)
Their perception is very likely to be totally different.

* They might not perceive some substances at all, others that we don't notice might make it unpalatable.

* Some substances might be perceived differently than us, or be indistinguishable from others.

* And some might require getting used to.

Note that all of the above phenomena also occur in humans because of genetics, cultural background, or experiences!

Re:LLaMA Now Goes Faster on CPUs by skeledrew on Monday April 01, 2024 @04:37PM (#39898979)
This may come off as pedantic, but "identical" is a very strong term when it comes to something like chemistry. The smallest chemical difference can manifest as a large physical difference. Consider that genetically, humans are about 60% similar to the fruit fly, yet phenotically, the similarity could be considered under 1%.
Re:LLaMA Now Goes Faster on CPUs by dekhn on Monday April 01, 2024 @03:32PM (#39898153)
Re:LLaMA Now Goes Faster on CPUs by skeledrew on Monday April 01, 2024 @09:36PM (#39901457)
An interesting thought experiment, but there's a flaw in it, an implicit fallacy that's probably a straw man. On its own, the argument would likely stand that Mary gains new knowledge on actually being exposed to color.

However, there is a broader context: this is supporting an argument against physicalism, and in this light it falls apart. There are a couple of missing bits required to complete the experiment in this context. The understanding that knowledge comes in 2 varieties: direct (actual experience) and indirect (description by one with the actual experience using shared language). This understanding brings proper clarity to the original argument, as we are aware - I think - that language is used to create compressed representations of things; something like a perceptual hash function.

The other key bit, which I guess we've only considered and extensively explored after the argument was formulated, is that all information coming in via the senses goes to the brain as electrical signals. And we actually have experimental data showing that sensory information can be emulated using machines. Thus, the original argument, to be relevant to the context, should be completed by giving Mary access to a machine that she can program to emulate the electrical signals that represent color experience.

I posit that without access to that hypothetical machine, given the context of the experiment, it cannot be said that Mary has "learned everything there is to learn about color". And once she has comprehensively and correctly utilized said machine on herself, she will gain no new knowledge when she is exposed to the world of color. Therefore this experiment cannot be used as an argument against physicalism as originally intended.

Re:LLaMA Now Goes Faster on CPUs by dekhn on Tuesday April 02, 2024 @12:58PM (#39908035)
Personally I don't think these complicated thought questions based on subjective experience enlighten us at all.
Re:LLaMA Now Goes Faster on CPUs by skeledrew on Tuesday April 02, 2024 @08:33PM (#39912395)
OK. I enjoyed the mental exercise though, thanks for that. Also, as someone who's formally studied philosophy, I'd say there is definitely value in thought experiments, particularly as I think we got to an objective level in this case, though we started with the subjective. And determining universal (objective) rules are valued as they usually help guide us to truth, and/or point to ideals to strive for.
Re:LLaMA Now Goes Faster on CPUs by visarga on Monday April 01, 2024 @05:42AM (#39892313)
I think we can safely say that any taste, smell, sensation or emotion of any importance has been described 1000 times over in the text corpus of GPT. Even though it is fragmented, by sheer volume there is enough signal in the training set, otherwise it would not be able to generate coherent text. In this case I think the map (language) is asymptotically close to the territory (sensations & experience in general).
Re:LLaMA Now Goes Faster on CPUs by andersa on Monday April 01, 2024 @02:04AM (#39891292)
What makes you think they aren't already?
Re:LLaMA Now Goes Faster on CPUs by wruza on Monday April 01, 2024 @01:12AM (#39891101)
That's where humans suck. The classic "you're not doing it right" then proceeds to quickly show how to do it without verbalizing any info on learning process, pitfalls, failure modes, etc, as if just showing it was enough for themselves to learn. Most people do[n't do] that, not even a sign of reflection.

My worst case was with a guy who asked me to write an arbitrage betting bot. When I asked how to calculate coeffs, he pointed at two values and said "look, there <x>, there <y> thinks for a minute then it's <z>!". When I asked how exactly did he calculate it, he simply repeated with different numbers.

Re:LLaMA Now Goes Faster on CPUs by Aerroon on Monday April 01, 2024 @02:39AM (#39891439)
People often don't know how to verbalize them in the first place. Some of these topics are very complex, but our intuition gets us halfway there.

Once upon a time I was good at a video game. Everyone realized that positioning is extremely important in this game.

I have good positioning in that game and was asked many times to make a guide about positioning. I never did, because I don't really know how. There is too much information they you need to convey to cover all the various situations.

I think you would first have to come up with a framework on positioning to be able to really teach this to someone else. Some kind of base truths/patterns that you can then use to convey the meaning. I believe the same thing applies to a lot of these processes that aren't verbalized.

Re:LLaMA Now Goes Faster on CPUs by snovv_crash on Monday April 01, 2024 @07:18AM (#39892897)
Often for this kind of problem writing a closed form solution is simply intractable. However, it's often still possible to express the cost function of at least a big portion of what goes into a human-optimal solution. From here you can sample your space, do gradient descent or whatever to find some acceptable solution that has a more human-intuitive property.
Re:LLaMA Now Goes Faster on CPUs by michaelt on Monday April 01, 2024 @08:13AM (#39893285)
It's not necessarily that it's intractable - just that a thing can be very hard to describe, under some circumstances.

Imagine someone learning English has written "The experiment reached it's conclusion" and you have to correct their grammar. Almost any english speaker can correct "it's" to "its" but unless they (and the person they're correcting) know a bunch of terms like 'noun' and 'pronoun' and 'possessive' they'll have a very hard time explaining why.

Re:LLaMA Now Goes Faster on CPUs by wruza on Wednesday April 03, 2024 @07:11PM (#39924526)
They may not even know why and it may be okay -- they speak it somehow, right? In this case, the language is both a set or rules and a systematization of a pre-existing phenomenon. There's enough ephemeral, hard to explain concepts, but most humans just aren't used to explain it even to themselves.

For example, I've never learned English, anywhere. I know it from .txt and .ng documents and a couple of dictionaries I had back in the DOS days. I'm an uneducated text-native, basically. But here's what I'd say to that newbie:

- Usually we use "...'s" for attribution like in "human's cat = cat of a human". But "it" and other special words like "that", "there", etc are an exception. We write "it's" as short for "it is", sometimes "it has". But we write "its", "theirs" for attribution, like in "its paw" = "paw of it" ~~ "cat's paw" = "paw of a cat". There's more to this, but you can ignore it for now.

Re:LLaMA Now Goes Faster on CPUs by samus on Monday April 01, 2024 @02:29AM (#39891388)
> When I asked how exactly did he calculate it, he simply repeated with different numbers.

Now you know how an LLM feels during training!

Re:LLaMA Now Goes Faster on CPUs by stavros on Monday April 01, 2024 @05:14AM (#39892114)
Probably during inference, as well.
Re:LLaMA Now Goes Faster on CPUs by Shorel on Monday April 01, 2024 @10:13AM (#39894379)
I wouldn't say this is where humans suck. On the contrary, this how we find human language is such a fantastic tool to serialize and deserialize human mental processes.

Language is so good, that an artificial language tool, without any understanding of these mental processes, can appear semi-intelligent to us.

A few people unable to do this serialization doesn't mean much on the larger scale. Just that their ideas and mental processes will be forgotten.

Re:LLaMA Now Goes Faster on CPUs by HarHarVeryFunny on Monday April 01, 2024 @11:00AM (#39894821)
> I contend that most human knowledge is not written down

Yes - the available training data is essentially mostly a combination of declarative knowledge (facts - including human-generated artifacts) and procedural knowledge (how to do things). What is missing is the learning process of taking a description of how to do something, and trying to apply that yourself in a specific situation.

No amount of reading books, or reading other people's blogs on how they did something, can avoid the need for hands-on experience if you want to learn how to do it yourself.

It's not just a matter of information that might be missing or unclear in instructional material, including how to cope with every type of failure and unexpected outcome, but crucially how to do this yourself - if you are to be the actor, then it's the predictive process in your mind that matters.

Partly for this reason, and partly because current AI's (transformer-based LLMs) don't support online learning (try & fail skill acquisition), I think we're going to see two distinct phases of AI.

1) The current "GenAI" phase where AI can only produce mash-ups of things it saw in it's pre-training data, augmented by similar "book learning" provided in-context which can be utilized by in-context learning. I'd characterize what this type of AI to be useful for, and capable of, as "automation". Applying that book (incl. anecdotal) knowledge to new situations where mash-up is all you need.

2) The second phase is where we have something closer to AGI, even if still below human level, which is no longer just a pre-trained transformer, but also has online learning and is agentic - taking actions predicated on innate traits like curiosity and boredom, so that given the book knowledge it can (& will!) then learn to apply that by experimentation/practice and learning from its own mistakes.

There will no doubt be advances beyond this "phase two" as well, but it seems we're likely to be stuck at "phase one" for a while (even as models become much better at phase one capabilities), until architectures fundamentally advance beyond transformers to allow this type of on-the-job training and skill acquisition.

Re:LLaMA Now Goes Faster on CPUs by mikewarot on Sunday March 31, 2024 @11:04PM (#39890546)
I don't see LLMs as a large chunk of knowledge, I see them as an emergent alien intelligence snapshotted at the moment it appeared to stop learning. It's further hobbled by the limited context window it has to use, and the probabilistic output structure that allows for outside random influences to pick its next word.

Both the context window and output structure are, in my opinion, massive impedance mismatches for the emergent intellect embedded in the weights of the model.

If there were a way to match the impedance, I strongly suspect we'd already have AGI on our hands.

Re:LLaMA Now Goes Faster on CPUs by namarie on Sunday March 31, 2024 @11:47PM (#39890738)
I can agree on the context windows, but what other output structure would you have?
Re:LLaMA Now Goes Faster on CPUs by moffkalast on Monday April 01, 2024 @02:38PM (#39897527)
Working with pure bytes is one option that's being researched. That way you're not really constrained by anything at all. Sound, images, text, video, etc. Anything goes in, anything comes out. It's hard to say if it's feasible with current compute yet without tokenizers to reduce dimensionality.
Re:LLaMA Now Goes Faster on CPUs by bamboozled on Sunday March 31, 2024 @11:23PM (#39890657)
What is alien about them ?

LLMs are of this earth and created by our species. Seems quite familiar to me.

Re:LLaMA Now Goes Faster on CPUs by jfoster on Sunday March 31, 2024 @11:50PM (#39890748)
They can write in a way similar to how a human might write, but they're not human.

The chat interfaces (Claude, ChatGPT) certainly have a particular style of writing, but the underlying LLMs are definitely capable of impersonating as our species in the medium of text.

Re:LLaMA Now Goes Faster on CPUs by bamboozled on Monday April 01, 2024 @07:03AM (#39892771)
But they're extremely relatable to us because it's regurgitating us.

I saw this talk with Geoffrey Hinton the other day and he said he was astonished at the capabilities of ChatGPT-4 because he asked it what the relationship between a compost heap and a nuclear bomb was, and he couldn't believe it answered, he really thought it was proof the thing could reason. Totally mind blown.

However I got it right away with zero effort.

Either I'm a super genius or this has been discussed before and made it's way into the training data.

Usual disclaimer: I don't think this invalidates the usefulness of AI or LLMs, just that we might be bamboozling ourselves into the idea that we've created an alien intelligence.

Re:LLaMA Now Goes Faster on CPUs by EMM_386 on Monday April 01, 2024 @01:58PM (#39897039)
> Either I'm a super genius or this has been discussed before and made it's way into the training data.

If an LLM can tell you the relatonship between a compost heap and nuclear bomb, that doesn't mean that was in the training data.

It could be because a compost heap "generates heat", and a nuclear bomb also "generates heat" and due to that relationship they have something in common. The model will pick up on these similar patterns. They tokens are positioned closer to each other in the high dimensional vector space.

But for any given "what does x have in common with y", that doesn't necessarily mean someone has asked that before and it's in the training data. Is that reasoning? I don't know ... how does the brain do it?

Re:LLaMA Now Goes Faster on CPUs by bamboozled on Monday April 01, 2024 @03:35PM (#39898195)
I mean that’s what sucks about Open AI isn’t it ? They won’t tell us what is in the training data so we don’t know. All I’m saying is that it wouldn’t be surprising if this was discussed previously somewhere in a pop science book.

That answer was close btw !

Re:LLaMA Now Goes Faster on CPUs by moffkalast on Monday April 01, 2024 @02:42PM (#39897587)
> how does the brain do it?

It's a lot of organic matmuls. ;)

Re:LLaMA Now Goes Faster on CPUs by jfoster on Tuesday April 02, 2024 @08:03AM (#39904782)
We used to have a test (Turing test) that could quite reliably differentiate between AI and our own species over the medium of text. As of now, we do not seem to have a simple & reliable test like that anymore.
Re:LLaMA Now Goes Faster on CPUs by fragmede on Monday April 01, 2024 @12:38AM (#39890965)
They don't think, they don't reason, they don't understand. Except they do. But it's hard for human words for thought processes to apply when giving it an endless string of AAAAA's makes it go bananas.

That's not familiar behavior. Nor is the counting reddit derived output. It's also not familiar for a single person to have the breadth and depth of knowledge that ChatGPT has. Sure, some people know more than others, but even without hitting the Internet, it has a ridiculous amount of knowledge, far surpassing a human, making it, to me, alien. though, it's inability to do math sometimes is humanizing to me for some reason.

ChatGPT's memory is also unhuman. It has a context window which is a thing, but also it only knows about things you've told it in each chat. Make a new chat and it's totally forgotten the nickname you gave it.

I don't think of HR Geiger's work, though made by a human, as familiar to me. it feels quite alien to me, and it's not just me, either. Dali, Bosch, and Escher are other human artists who's work can be unfamiliar and alien. So being created by our species doesn't automatically imbue something with familiar human processes.

So it dot products, it matrix multiplies, instead of reasoning and understanding. It's the Chinese room experiment on steroids; it turns out a sufficiently large corpus on a sufficiently large machine does make it look like something"understands".

Re:LLaMA Now Goes Faster on CPUs by samus on Monday April 01, 2024 @03:13AM (#39891582)
The context window is comparable to human short-term memory. LLMs are missing episodic memory and means to migrate knowledge between the different layers and into its weights.

Math is mostly impeded by the tokenization, but it would still make more sense to adapt them to use RAG to process questions that are clearly calculations or chains of logical inference. With proper prompt engineering, they can process the latter though, and deviating from strictly logical reasoning is sometimes exactly what we want.

The ability to reset the text and to change that history is a powerful tool! It can make the model roleplay and even help circumvent alignment.

I think that LLMs could one day serve as the language center of an AGI.

Re:LLaMA Now Goes Faster on CPUs by trimethylpurine on Monday April 01, 2024 @02:10AM (#39891318)
The word "alien" works in this context but, as the previous commenter mentioned, it also carries the implication of foreign origin. You could use "uncanny" instead. Maybe that's less arbitrary and more specific to these examples.

"Alien" still works, but then you might have to add all the context at length, as you've done in this last comment.

Re:LLaMA Now Goes Faster on CPUs by fire_lake on Monday April 01, 2024 @02:37AM (#39891423)
Hype people do this all the time - take a word that has a particular meaning in a narrow context and move it to a broader context where people will give it a sexier meaning.

    AI researchers unveil alien intelligence
Is way better headline.
Re:LLaMA Now Goes Faster on CPUs by taneq on Monday April 01, 2024 @03:37AM (#39891684)
In all fairness, going up to SMS random human and yelling AAAAAAAAAAAAAA… at them for long enough will produce some out-of-distribution responses too.
Re:LLaMA Now Goes Faster on CPUs by cloudwalk9 on Monday April 01, 2024 @11:15AM (#39895016)
Makes me think that TikTok and YT pranksters are accidentally producing psychological data on what makes people tick under scenarios of extreme deliberate annoyance. Although the quality (and importance) of that data is obviously highly variable and probably not very high, and depends on what the prank is.
Re:LLaMA Now Goes Faster on CPUs by inference-lord on Monday April 01, 2024 @08:19AM (#39893336)
Do you find a large database or spreadsheet that hold more information than you can "alien" too?
Re:LLaMA Now Goes Faster on CPUs by mikewarot on Monday April 01, 2024 @06:06PM (#39899970)
Alien meaning unfamiliar, not necessarily extraterrestrial.

Aliens are people from other countries, for example.

Exotic would be another good word to use.

Re:LLaMA Now Goes Faster on CPUs by mlsu on Monday April 01, 2024 @03:53AM (#39891739)
Disagree. The input/output structure (tokens) is the interface for both inference and for training. There is an emergent intellect embedded in the weights of the model. However, it is only accessible through the autoregressive token interface.

This is a fundamental limitation, much more fundamental than appears at first. It means that the only way to touch the model, and for the model to touch the world, is through the tokenizer (also, btw, why tokenizer is so essential to model performance). Touching the world through a tokenizer is actually quite limited.

So there is an intelligence in there for sure, but it is locked in an ontology that is tied to its interface. This is even more of a limitation than e.g. weights being frozen.

Re:LLaMA Now Goes Faster on CPUs by creatonez on Monday April 01, 2024 @12:06AM (#39890818)
Maybe I'm seeing things through a modern lens, but if I were trying to restart civilization and was only left with ChatGPT, I would be enraged and very much not grateful for this.
Re:LLaMA Now Goes Faster on CPUs by nyokodo on Monday April 01, 2024 @01:53AM (#39891251)
> if I were trying to restart civilization and was only left with ChatGPT

In this scenario you’d need to also be left with a big chunk of compute, and power infrastructure. Since ChatGPT is the front end of the model you’d also need to have the internet still going in a minimum capacity.

Re:LLaMA Now Goes Faster on CPUs by CaptainOfCoit on Monday April 01, 2024 @10:53AM (#39894754)
If we're playing this game, you forgot to mention that they also need: A monitor, a keyboard, roof over their head (to prevent rain from entering your electronic), etc etc...

But really, didn't you catch the meaning of parents message, or are you being purposefully obtuse?

Re:LLaMA Now Goes Faster on CPUs by devsda on Monday April 01, 2024 @04:41AM (#39891955)
I think re-imagining the "Dr. Stone" series with the main character replaced by an LLM will be a funny & interesting series if we decide to stay true to LLMs nature and make it hallucinate as well.

Given the way LLMs are right now, I suspect there will be lot of failed experiments and the kingdom of science will not advance that quick.

Re:LLaMA Now Goes Faster on CPUs by latexr on Monday April 01, 2024 @08:06AM (#39893233)
> the kingdom of science will not advance that quick.

It’s more likely that it wouldn’t even start. The first step to any development was figuring out nitric acid as the cure to the petrification. Good luck getting any LLM to figure that out. Even if it did, good luck getting any of the other characters to know what to do with that information that early on.

Re:LLaMA Now Goes Faster on CPUs by jrflowers on Sunday March 31, 2024 @11:47PM (#39890739)
It is invaluable to have a chunk of human knowledge that can tell you things like the Brooklyn Nets won the 1986 Cricket World Cup by scoring 46 yards in only 3 frames
Re:LLaMA Now Goes Faster on CPUs by fragmede on Monday April 01, 2024 @12:42AM (#39890981)
According to ChatGPT

> Australia won the 1987 Cricket World Cup. The 1986 date is incorrect; there was no Cricket World Cup in 1986. The tournament took place in 1987, and Australia defeated England in the final to win their first title.

https://chat.openai.com/share/e9360faa-1157-4806-80ea-563489...

I'm no cricket fan, so someone will have to correct Wikipedia if that's wrong.

If you want to point out that LLMs hallucinate, you might want to speak plainly and just come out and say it, or at least give a real world example and not one where it didn't.

Re:LLaMA Now Goes Faster on CPUs by vlunkr on Monday April 01, 2024 @01:18AM (#39891129)
We’re not talking about running chatGPT locally though, are we?
Re:LLaMA Now Goes Faster on CPUs by fragmede on Monday April 01, 2024 @01:22AM (#39891147)
sigh your going to make me open my laptop, aren't you.
Re:LLaMA Now Goes Faster on CPUs by fragmede on Monday April 01, 2024 @01:36AM (#39891197)
I ran 'who won the 1986 Cricket World Cup' against llama2-uncensored (the local model I have pre-downloaded) and hilarious got 5 different answers asking it 5 times:

    >>> who won the 1986 Cricket World Cup
    India
    
    >>> who won the 1986 Cricket World Cup
    Australia
    
    >>> who won the 1986 Cricket World Cup
    New Zealand
    
    >>> who won the 1986 Cricket World Cup
    West Indies
    
    >>> who won the 1986 Cricket World Cup
    England
Which proves GP's point about hallucinations, though none of those are

> Brooklyn Nets won the 1986 Cricket World Cup by scoring 46 yards in only 3 frames

LLM's hallucinations are insidous because they have the ring of truth around them. yards and frames aren't cricket terms, so we're off to the races with them.

Re:LLaMA Now Goes Faster on CPUs by astrange on Monday April 01, 2024 @02:24AM (#39891365)
If you want factual answers from a local model it might help to turn the temperature down.
Re:LLaMA Now Goes Faster on CPUs by fragmede on Monday April 01, 2024 @04:05AM (#39891792)
It would also help if I had more VRAM and wasn't running a 7B parameter 4-bit quantized model.
Re:LLaMA Now Goes Faster on CPUs by jrflowers on Monday April 01, 2024 @02:43AM (#39891455)
> If you want factual answers from a local model it might help to turn the temperature down.

This makes sense. If you interact with a language model and it says something wrong it is your fault

Re:LLaMA Now Goes Faster on CPUs by astrange on Monday April 01, 2024 @04:52AM (#39892011)
You're not "interacting with a language model", you're running a program (llama.cpp) with a sampling algorithm which is not set to maximum factualness by default.

It's like how you have to set x264 to the anime tuning or the film tuning depending on what you run it on.

Re:LLaMA Now Goes Faster on CPUs by beefnugs on Monday April 01, 2024 @03:13AM (#39891587)
Actually isn't this good? It means we can run something multiple times to prove itself a bad answer?
Re:LLaMA Now Goes Faster on CPUs by latexr on Monday April 01, 2024 @08:15AM (#39893299)
You can ask LLMs the same question and they might sometimes get it wrong and other times get it right. Having different answers is no indication that none of them is correct.

Furthermore, even if an LLM always gives the same answer to a question, there’s no guarantee the answer is correct.

https://en.wikipedia.org/wiki/Propaganda

https://en.wikipedia.org/wiki/Big_lie#Alleged_quotation

Re:LLaMA Now Goes Faster on CPUs by sroussey on Monday April 01, 2024 @01:45PM (#39896847)
An LLM will always give the same output for the same input. It’s sorta like a random number generator that gives the same list of “random” numbers for the same seed. LLMs get a seed too.
Re:LLaMA Now Goes Faster on CPUs by latexr on Monday April 01, 2024 @04:46PM (#39899109)
That’s irrelevant for the matter. The person I replied to obviously did not have seeded responses in mind.
Re:LLaMA Now Goes Faster on CPUs by saagarjha on Tuesday April 02, 2024 @11:56PM (#39913465)
It can tell you if the answer is wrong, but it can never tell you if the answer is right.
Re:LLaMA Now Goes Faster on CPUs by ilaksh on Monday April 01, 2024 @09:26AM (#39893952)
You should specify the model size and temperature.

For fact retrieval you need to use temperature 0.

If you don't get the right facts then try 34b, 70b, Mixtral, Falcon 180b, or another highly ranked one that has come out recently like DBRX.

Re:LLaMA Now Goes Faster on CPUs by samus on Monday April 01, 2024 @02:49AM (#39891479)
The facts LLMs learned from training are fuzzy, unreliable, and quickly outdated. You actually want retrieval-augmented generation (RAG) where a model queries an external system for facts or to perform calculations and postprocesses the results to generate an answer for you.
Re:LLaMA Now Goes Faster on CPUs by unshavedyak on Monday April 01, 2024 @10:37AM (#39894596)
Is there a name for the reverse? I'm interested in having a local LLM monitor an incoming, stateful data stream. Imagine chats. It should have the capability of tracking the current day, active participants, active topics, etc - and then use that stateful world view to associate metadata with incoming streams during indexing.

Then after all is indexed you can pursue RAG on a richer set of metadata. Though i've got no idea what that stateful world view is.

Re:LLaMA Now Goes Faster on CPUs by firewolf34 on Tuesday April 02, 2024 @03:52AM (#39903307)
This is an interesting idea but I'm having trouble understanding what you're to achieve. Do you mean the LLM would simply continuously update it's context window with incoming data feeds realtime, and you use it as an interface? That's pretty akin to summarization task, yes? Or are you augmenting the streams with "metadata" you mentioned?
Re:LLaMA Now Goes Faster on CPUs by unshavedyak on Tuesday April 02, 2024 @11:10AM (#39906616)
Yea, the state i mentioned i think would be managed by several entities. Ie time, current date, etc - all could be automated without involvement of the LLM of course. However as conversations come in, the LLM would also modify the state with context clues from the conversation.

Then, when future messages come in from alternate streams (say, browser history), they could (maybe, hah) be made more rich. More likely though i would expect it to be the opposite scenario, browser informs chat, etc.

I say this because in many cases i imagine my chat conversations in my household have a severe lack of context. We often jump to vocal communication, and then paste links, etc. In a perfect world i think i'd even take home camera audio transcripts and do the same.

Ie i don't want to _just_ index a browser log as "interested in Rust. Some library about BTree", etcetc - but additional sources of data could try to store what it is i am actively doing, and associate that to the browser log.

All of this of course is nothing i'd ever want to leave the house. My hope though is that it would lean into what LLMs do well. Without the expectation of actual LLM intelligence.

Re:LLaMA Now Goes Faster on CPUs by firewolf34 on Wednesday April 10, 2024 @01:40AM (#39987314)
So perhaps you're suggesting we sort of "boil-down" an information source into a sort of base representation of meaning and intent, something similar to vector store, and relate the many inputs together in this space using the LLM as glue like one does manually creating links in a web of Zettelkasten for research. I think this is something that the field is rapidly moving towards in personal information management.
Re:LLaMA Now Goes Faster on CPUs by dolmen on Tuesday April 02, 2024 @08:35AM (#39905001)
LLMs don't learn facts.

They learn statistics on texts and are able to regurgitate them somewhat.

Re:LLaMA Now Goes Faster on CPUs by TheCaptain4815 on Sunday March 31, 2024 @11:55PM (#39890772)
It’s kind of crazy really. Before LLMs, any type of world scale disaster you’d hope for what? Wikipedia backups? Now, a single LLM ran locally would be much more effective. Imagine the local models in 5 years!
Re:LLaMA Now Goes Faster on CPUs by danmur on Monday April 01, 2024 @12:24AM (#39890909)
Uh yeah I would, and still am, take the Wikipedia backup for doomsday scenarios. I'm not even sure how that would be a competition
Re:LLaMA Now Goes Faster on CPUs by Zambyte on Monday April 01, 2024 @01:02AM (#39891069)
The processing required to run current language models with a useful amount of knowledge encoded in them is way more than I imagine would be available in a "world scale disaster".
Re:LLaMA Now Goes Faster on CPUs by int_19h on Monday April 01, 2024 @02:44AM (#39891460)
There's a lot more than just Wikipedia that gets archived, and yes, that is a far more sensible way to go about it. For one thing, the compute required to then read it back is orders of magnitude less (a 15 year old smartphone can handle it just fine). For another, you don't have to wonder how much of what you got back is hallucinated - data is either there or it's corrupted and unreadable.
Re:LLaMA Now Goes Faster on CPUs by gpm on Sunday March 31, 2024 @11:16PM (#39890615)
If you want to download a backup of a large chunk of human knowledge... download wikipedia. It's a similar size to a small LLM and can actually distinguish between real life and fantasy: https://en.wikipedia.org/wiki/Wikipedia:Database_download

If you just want to play around with an LLM though, absolutely.

Re:LLaMA Now Goes Faster on CPUs by int_19h on Monday April 01, 2024 @02:41AM (#39891447)
Kiwix provides prepackaged highly compressed archives of Wikipedia, Project Gutenberg, and many other useful things: https://download.kiwix.org/zim/.

Between that and dirt cheap storage prices, it is possible to have a local, offline copy of more human knowledge than one can sensibly consume in a lifetime. Hell, it's possible to have it all on one's smartphone (just get one with an SD card slot and shove a 1+ Tb one in there).

Re:LLaMA Now Goes Faster on CPUs by claritise on Monday April 01, 2024 @08:38AM (#39893498)
Just create a RAG with wikipedia as the corpus and a low parameter model to run it and you can basically have an instantly queryable corpus of human knowledge runnable on an old raspberry pi.
Re:LLaMA Now Goes Faster on CPUs by CaptainOfCoit on Monday April 01, 2024 @10:06AM (#39894325)
> a low parameter model

> on an old raspberry pi

I bet the LLM responses will be great... You're better off just opening up a raw text dump of Wikipedia markup files in vim.

Re:LLaMA Now Goes Faster on CPUs by boywitharupee on Monday April 01, 2024 @10:20AM (#39894423)
but which model to tokenize with? is there a leaderboard for models that are good for RAG?
Re:LLaMA Now Goes Faster on CPUs by sroussey on Monday April 01, 2024 @01:36PM (#39896727)
“For RAG” is ambiguous.

First there is a leaderboard for embeddings. [1]

Even then, it depends how you use them. Some embeddings pack the highest signal in the beginning so you can truncate the vector, while most can not. You might want that truncated version for a fast dirty index. Same with using multiple models of differing vector sizes for the same content.

Do you preprocess your text? There will be a model there. Likely the same model you would use to process the query.

There is a model for asking questions from context. Sometimes that is a different model. [2]

Re:LLaMA Now Goes Faster on CPUs by Workaccount2 on Monday April 01, 2024 @11:17AM (#39895040)
Pretty neat to have laying around, thanks
Re:LLaMA Now Goes Faster on CPUs by CaptainOfCoit on Monday April 01, 2024 @10:03AM (#39894304)
> actually distinguish between real life and fantasy

Are LLMs unable to distinguish between real life and fantasy? What prompts have you thrown at them to make this determination? Sending a small fairy tale and asking the LLM if it thinks it's a real story or fake one?

Re:LLaMA Now Goes Faster on CPUs by gpm on Monday April 01, 2024 @10:27AM (#39894497)
... having them talk about events from sci fi stories in response to questions about the real world. Having them confidently lie about pretty much everything. Etc.
Re:LLaMA Now Goes Faster on CPUs by CaptainOfCoit on Monday April 01, 2024 @10:51AM (#39894731)
What are the specific prompts you're using? You might get those answers when you're not being specific enough (or use models that aren't state of the art).

"Shit in, shit out" as the saying goes, but applied to conversations with LLMs where the prompts often aren't prescriptive enough.

Re:LLaMA Now Goes Faster on CPUs by texuf on Sunday March 31, 2024 @10:59PM (#39890507)
Any recommendations for the latest and greatest way to run these locally?
Re:LLaMA Now Goes Faster on CPUs by etc-hosts on Sunday March 31, 2024 @11:10PM (#39890577)
Re:LLaMA Now Goes Faster on CPUs by threecheese on Monday April 01, 2024 @06:37PM (#39900222)
This looks amazing, but the docs mention .llamafiles exceed the Windows executable size limit, and there are workarounds to externalize the weights. Do you think this is an impediment to its becoming popular? Or is MS consumer hardware just far enough behind (w/o dedi gpu) that “there’s time”?
Re:LLaMA Now Goes Faster on CPUs by speps on Sunday March 31, 2024 @11:00PM (#39890512)
llamafile as per TFA...
Re:LLaMA Now Goes Faster on CPUs by fragmede on Monday April 01, 2024 @12:45AM (#39891001)
ollama
Re:LLaMA Now Goes Faster on CPUs by slowmotiony on Monday April 01, 2024 @04:05AM (#39891794)
I use a tool called LM Studio, makes it trivial to run these models on a Mac. You can also use it as a local API so it kinda acts like a drop-in replacement for the openAI API.
Re:LLaMA Now Goes Faster on CPUs by chown on Monday April 01, 2024 @09:16AM (#39893859)
I am the author of Msty [1]. My goal is to make it as straightforward as possible with just one click (once you download the app). If you end up trying it, I would love to hear your feedback.

1: https://msty.app

Re:LLaMA Now Goes Faster on CPUs by m3kw9 on Monday April 01, 2024 @12:47AM (#39891011)
And why would I need to backup human knowledge as an individual
Re:LLaMA Now Goes Faster on CPUs by exe34 on Monday April 01, 2024 @05:37AM (#39892285)
You remember those fantasies where you got up from your seat at the pub and punched the lights out of this guy for being rude? A lot of us have fantasies of being the all powerful oracle that guides a reboot of civilization using knowledge of science and engineering.
Re:LLaMA Now Goes Faster on CPUs by latexr on Monday April 01, 2024 @08:18AM (#39893323)
> the all powerful oracle that guides a reboot of civilization using knowledge of science and engineering.

https://en.wikipedia.org/wiki/Dr._Stone

Re:LLaMA Now Goes Faster on CPUs by raincole on Monday April 01, 2024 @01:53AM (#39891250)
It seems to be an unbelievably inefficient way to back up knowledge.
Re:LLaMA Now Goes Faster on CPUs by samus on Monday April 01, 2024 @03:27AM (#39891642)
Are they though? They are lossy compressing trillions of tokens into a few dozen GB. The decompression action is fuzzy and inefficient though.
Re:LLaMA Now Goes Faster on CPUs by raincole on Monday April 01, 2024 @04:16AM (#39891846)
And it requires massive computational power to decompress, which I don't expect to be available in a catastrophic situation where humans have lost a large chunk of important knowledge.
Re:LLaMA Now Goes Faster on CPUs by samus on Monday April 01, 2024 @04:39AM (#39891945)
I don't necessarily agree. It requires massive computing power, but running models smaller than 70G parameters is possible on consumer hardware, albeit slowly.
Re:LLaMA Now Goes Faster on CPUs by threecheese on Monday April 01, 2024 @06:31PM (#39900176)
Parent may be thinking more along the lines of a “hope we can print all the knowledge“ type catastrophe. Though if there is zero compute it’ll be tough reading all those disks!
Re:LLaMA Now Goes Faster on CPUs by LunaSea on Monday April 01, 2024 @04:35AM (#39891923)
I wonder how the Chinese government will manage to sensor LLMs within China?
Re:LLaMA Now Goes Faster on CPUs by popol12 on Monday April 01, 2024 @07:08AM (#39892822)
The same way Facebook/Google/openAI & others censored their own LLMs, I guess ?
Re:LLaMA Now Goes Faster on CPUs by LunaSea on Monday April 01, 2024 @09:09AM (#39893772)
That's only for SaaS LLMs, but if you can simply download and run one on your hardware, things become difficult.
Re:LLaMA Now Goes Faster on CPUs by kalleboo on Monday April 01, 2024 @06:25AM (#39892548)
I had downloaded some LLMs to run locally just to experiment when a freak hailstorm suddenly left me without internet for over a week. It was really interesting to use a local LLM as a replacement for Google.

It gave me a new mental model for LLMs rather than a "spicy autocomplete" or whatever, I now think of it as "a lossy compressed database of knowledge". Like you ran the internet through JPEG at 30% quality.

Re:LLaMA Now Goes Faster on CPUs by pizzafeelsright on Monday April 01, 2024 @10:57AM (#39894791)
Feels like that really smart friend who is probably correct but ya just don't know.
Re:LLaMA Now Goes Faster on CPUs by dragonwriter on Monday April 01, 2024 @01:44PM (#39896840)
Language models are an inefficient way to store knowledge; if you want to have a “pseudo-backup of a large chunk of human knowledge,” download a wikipedia dump, not an LLM.

If you want a friendly but fallible UI to that dump, download an LLM and build a simple ReAct framework around it with prompting to use the wikipedia dump for reference.

Re:LLaMA Now Goes Faster on CPUs by TrevorJ on Monday April 01, 2024 @05:12PM (#39899411)
It's a very underrated side effect of this whole LLM thing: We've created a super compact representation of human knowledge in a form that requires a FAR less complex tech stack to get the information 'out' of in the future.

A year ago, a lot of this information only existed on the internet, and would have been nearly impossible to recover in any cohesive unfragmented form if the lights were to ever go out on our civilization.

Now the problem space has moved simply to "find a single solitary PC that will still boot up", and boom, you have access to everything.

I think we just created our Rosetta stone.