News for hackers. Stuff that matters.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by swyx on Monday March 13, 2023 @04:19PM (#35141904)
I feel like there has to be another shoe to drop here, this seems almost too good to be true.

> Alpaca shows that you can apply fine-tuning with a feasible sized set of examples (52,000) and cost ($600) such that even the smallest of the LLaMA models—the 7B one, which can compress down to a 4GB file with 4-bit quantization—provides results that compare well to cutting edge text-davinci-003 in initial human evaluation.

this is the most exciting thing. the cost of finetuning is rapidly coming down which means everyone will be able to train their own models for their usecases.

Looking for the contrarians on HN: what is being left unsaid here that people like myself and Simon might be getting too optimistic about? what are the known downsides that people in academia already know about?

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by porcc on Monday March 13, 2023 @04:25PM (#35141978)
We saw this happen with Stable Diffusion and it's not surprising we see this happening here. There is a lot of interest in taking these models that are in striking distance (single order of magnitude) from running inference and training on consumer level hardware and as such a lot of energy is going into making the optimizations that can get us there.

Generally speaking, research is not usually done with consumer usage in mind, so what this is, and Dreambooth etc. for Stable Diffusion was, is that gap between researcher software and accessible software being bridged.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by atleastoptimal on Monday March 13, 2023 @04:37PM (#35142175)
"initial human evaluation" is codeword for "cherrypicked prompts given to people who don't know how to trick an LLM"
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by sebzim4500 on Monday March 13, 2023 @05:10PM (#35142717)
I don't understand why that is a bad thing. If your goal is to make an AI assistant, then you should be optimizing for giving answers that real users find useful, not trying to impress other AI researchers.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by ChubbyGlasses on Monday March 13, 2023 @05:12PM (#35142746)
i always found this to be a strange pov to have on LLMs. imo, it's not humans tricking/gaming the ai, but rather chatgpt has tricked you into believing it's smarter than it actually is. (in human terms, chatgpt is just more articulate than llama)

it's a subtle distinction, but i think it shapes and reflects how you view ai as a tool for humans or as a replacement.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by bitL on Monday March 13, 2023 @10:59PM (#35146819)
There aren't any good automated metrics for generative LLM quality, so it's all up to humans to evaluate.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by smoldesu on Monday March 13, 2023 @04:29PM (#35142050)
> what is being left unsaid here that people like myself and Simon might be getting too optimistic about?

The past week has felt like a wake-up call to enthusiasts. Running models locally has been available for a while (even small, fairly coherent ones), and the majority of "improvements" recently have come from implementing the leaked LLaMa model.

The results from 7B are an improvement on what we had a year ago, but not by much. We're learning that there's room to optimize these models, but also that size matters. ChatGPT and 7B are both great at bullshitting, but you can feel the difference in model size during regular conversation. Adding insult to injury, it will almost always be faster to query an API for AI results than it will be to run it locally.

Analysis: Things are moving at a clip right now, but people expecting competitive LLMs running locally on their smartphone will be disappointed for quite a while. As the technology improves, it's also safe to assume that we'll find ways to scale model intelligence with greater resources, and the status quo will look much different than it does today.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by atleastoptimal on Monday March 13, 2023 @04:40PM (#35142218)
>API for AI results than it will be to run it locally. True, and remotely called APIs will always be the mover for the AI craze. Only niche hobbyists will be running them locally.

There is no company on the planet that would benefit from providing people local means to run LLMs. As a result only hacks and leaks will be how individuals can manage to run LLMs outside of heavily monitored remote API calls.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by mikek on Monday March 13, 2023 @04:52PM (#35142397)
Apple comes to mind.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by BryantD on Monday March 13, 2023 @05:49PM (#35143357)
Agreed. Apple will run models remotely if necessary, but from a PR perspective they align with their stated intentions when they can run locally.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by mustacheemperor on Monday March 13, 2023 @08:27PM (#35145441)
Oh, how I hope there is a project ongoing at Apple right now to implement some kind of on-device LLM that can drive the phone a la Siri. Any time I open the Bing app on my iPhone to answer a question I think of how useful it could be to ask it to update my calendar, fill out a dinner order, etc. How much time do we collectively waste putzing around with delivery apps etc?
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by smoldesu on Monday March 13, 2023 @08:48PM (#35145671)
It would be neat, but iPhone hardware has limited memory (and iOS limits that usage further). It would be much faster and higher-quality to run it as an API request like it is today.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by yieldcrv on Tuesday March 14, 2023 @02:33AM (#35148078)
Given the incentive, it can push the hardware in this direction.

Or they’ll do a home network device like implement it like a HomeKit hub. So the plugged in resource has more computational power and feeds it to iphones when nearby. Disconnected, the iphone uses a much more rudimentary one or falls back to Siri or a cloud service.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by gpderetta on Tuesday March 14, 2023 @04:40AM (#35148764)
Your phone can remotely connect to your home device when needed.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by smoldesu on Tuesday March 14, 2023 @10:29AM (#35152100)
Maybe it will use lasers to paint a smiley face on my wall!

Only two things are for certain; you'll have to pay for it, and it will not be cheap.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by gpderetta on Tuesday March 14, 2023 @04:39AM (#35148760)
Apple Siri as a device you can run on your home as a digital assistant.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by flangola7 on Monday March 13, 2023 @04:53PM (#35142406)
Who said anything about companies? Companies don't benefit by giving people free access to buildings full of books and knowledge, yet here they are.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by atleastoptimal on Monday March 13, 2023 @05:17PM (#35142835)
In the last 50 years of AI research has any academic institution ever provided open source easy to use tools like the stuff big companies have put out in the past 5 years?
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by simonw on Monday March 13, 2023 @05:33PM (#35143115)
Stable Diffusion came from an academic research lab.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by atleastoptimal on Monday March 13, 2023 @08:09PM (#35145247)
I thought stability AI was a private company
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by simonw on Monday March 13, 2023 @11:08PM (#35146887)
They are, but the team who trained Stable Diffusion are from academia. Stability sponsored the model training costs.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by flangola7 on Tuesday March 14, 2023 @01:18AM (#35147703)
As I'm aware neither had put out any such tools for 45 years
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by niemandhier on Monday March 13, 2023 @05:01PM (#35142549)
Companies like Facebook can harm their competitors by releasing models.

Facebook is not a major player in the Llm field, the technological advantage of openai is to large, BUT they can reduce the expected gains of their competition by providing less powerful alternatives for free.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by ALittleLight on Tuesday March 14, 2023 @04:29AM (#35148698)
Facebook may destroy their social networks by releasing LLMs. What will happen to spam when it could be driven by local LLMs? You'll have realistic accounts, with realistic pictures, posting realistic messages only to spam links or ads x% of the time.

(Though to be clear I do hope Facebook does release more models because I want to play with them)

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by smoldesu on Tuesday March 14, 2023 @10:31AM (#35152139)
I wouldn't count on it. However, I definitely think Facebook's remaining managers are scrambling to control this LLaMa leak though. They don't want the liability or responsibility of managing this project, and now everyone and their mother wants to play with it. Battling spam can largely be managed through the usual methods, but fighting a leaked model is much harder.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by Kye on Monday March 13, 2023 @04:24PM (#35141972)
The big failure mode is they can hallucinate nonsense that isn't obviously nonsense. You have to check any facts against expert sources. At that point, you could just email an expert who can use their own LLM to whip up an answer and check the facts themselves.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by simonw on Monday March 13, 2023 @04:55PM (#35142444)
That's a big problem if you're using a language model as a search engine. The trick is to learn how to use them for the things that they're good for outside of that.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by typest on Monday March 13, 2023 @05:08PM (#35142692)
^^ this. For instance, LLMs are really good at turning natural language into SQL. And if you know SQL, you can read it and make sure it looks good. But, it's much faster and easier than writing SQL by hand.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by flir on Monday March 13, 2023 @06:28PM (#35143951)
But that's still "You have to check any facts against expert sources"! You just have the advantage of being your own personal expert.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by hesk on Monday March 13, 2023 @09:21PM (#35146021)
Verifying a solution is easier than coming up with it.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by blueblimp on Monday March 13, 2023 @05:24PM (#35142978)
It seems still unclear how much quality loss there is compared to the best models. What's really needed is systematic evaluation of the output quality, but that's tricky and relatively expensive (compared to automated benchmarks), so I understand why it hasn't happened yet.

Edit: I just tried it with a single task of my own (that I've successfully used with ChatGPT and Bing) and it flubbed it horribly, so this model at least is noticeably inferior to the SOTA, which is not surprising given how small it is.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by yunyu on Monday March 13, 2023 @05:50PM (#35143393)
I assume you haven't tried Alpaca (which hasn't been released), only Llama. See the instruction fine tuning section in the article.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by karmasimida on Monday March 13, 2023 @06:16PM (#35143791)
They currently only supports a single input/response format of input right? Multi turns will be more challenging to handle.

I am optimistic for 30B or 66B to catch up with OpenAI, but 7B is unlikely to have the same quality.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by enlyth on Monday March 13, 2023 @07:06PM (#35144459)
Here is a sample chatbot style conversation I had locally with LLAMA 30B 4bit quantization on a 3090 RTX (generation speed 5 tokens per second):

https://i.imgur.com/V4lzLz7.png

It does quite well at simple reasoning. More complex stuff, it does struggle sometimes, but I am impressed by the output and the fact that it even runs on my computer.

I used the oobabooga repository on Ubuntu 22.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by plaba417 on Monday March 13, 2023 @07:06PM (#35144465)
I have had a few questions where it gave a completely nonsensical answer, but by fiddling with the parameters, I've gotten more usable output. Have you tried lowering the noise and increasing top_p?
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by flir on Monday March 13, 2023 @06:25PM (#35143918)
At a guess: assuming quality scales with size, the model in the data centre is always going to outcompete the model on the device. So in any situation where you've got bandwidth >4800bps, why would you choose the model on the device?
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by CuriouslyC on Monday March 13, 2023 @06:58PM (#35144347)
Your own fine tuning, no restrictions on output, privacy/security, and if you have a reason to produce a lot of output it'll be cheaper. Use ChatGPT if you only want to use it occasionally, you don't care about privacy/security in this context, the output restrictions don't bother you and having the best possible language model is the most important thing to you.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by zone411 on Monday March 13, 2023 @09:37PM (#35146151)
Right, not sending out your text to third parties is absolutely huge for certain industries. ChatGPT has been already banned by some companies, so the issue is whether or not the language model can be used at all.

I'll add the possibility of network-based service downtime to the list.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by dougmwne on Monday March 13, 2023 @06:31PM (#35143992)
First catch is that someone needed to spend the enormous up front cost to train the base model, then release it under a flexible enough license for your use case.

The second catch is that you would get much higher quality out of the 65b model, but would need to lay out a few thousand for the hardware.

The third catch is that you need the fine tuning data, but that seems easier than ever to create out of more capable LMMs.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by recuter on Monday March 13, 2023 @09:56PM (#35146321)
The 65b model now runs on the 14" MBP with 64GB of RAM for $3,500.

The base config is $2500 to begin with. So not thousands plural, it is one grand more than the base config and a lot of devs/pros go for the 32GB anyhow.

Apple laptops also have atypically high resale value, I usually sell mine after a couple of years for 50%+ of what I paid for it. By the by, Apple have a two week return policy. ;)

I think only your first catch applies. But if you can come up with the right mvp even that might fall away.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by Taek on Monday March 13, 2023 @10:14PM (#35146456)
Don't need a macbook anymore, any AVX2 CPU can do more than one token per second.

You could spend $600 and build your own PC that rubs LLaMA 65b.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by recuter on Monday March 13, 2023 @11:09PM (#35146906)
It is still much faster/cost-effective thanks to the unified memory. Would be interested in building out a PC as soon as something comparable becomes available.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by Taek on Monday March 13, 2023 @11:58PM (#35147276)
If you are going to spend $3500 on macbook because its more cost effective, might as well go all the way to buying an a6000 for $4095 which is iirc 5-10x faster than macbook
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by recuter on Tuesday March 14, 2023 @12:10AM (#35147343)
I am agnostic about Mac/PC but people sure like to chime in with these sort of comments.

First of all it is $4,700 on Newegg and secondly it is out of stock. This also leaves me with the rest of the computer to put together which is not $0. And finally it would absolutely not be 10x faster for this use case than a 64GB MBP.

The advantage of going with Apple is that I can pick one up immediately, use it as a laptop (soon with Asahi linux when it matriculates fully) and if need be easily sell it. You might not see the value in that, that's alright, people have different use cases and needs.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by dougmwne on Tuesday March 14, 2023 @08:36AM (#35150488)
How are you going to do training on that machine in a reasonable amount of time? Most of the utility of these models are unlocked after fine tuning.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by recuter on Tuesday March 14, 2023 @05:06PM (#35158623)
Through sheer tyranny of will!
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by bitL on Monday March 13, 2023 @10:57PM (#35146808)
Fine-tuning is both easy and cheap using transformer adapters, e.g. any Huggingface model.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by a_bonobo on Tuesday March 14, 2023 @01:32AM (#35147764)
>what are the known downsides that people in academia already know about?

I work in biology with some interface to wildlife managers/'end users' of our biological insights. I was hoping we could use these models as advanced chatbots so that our end users could ask biological questions from our data.

However, even the replies to the most basic biological questions in LLAMA/ChatGPT/OpenChatKit are hallucinated, even when setting temperature to 0. We simply can't trust any of these answers; the bot needs to be as truthful as humans in its replies, but none of them are so far. So what good are these models then?

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by int_19h on Tuesday March 14, 2023 @01:58AM (#35147906)
This is the wrong way to go about it. Instead of trying to force that informaiton into the model via training, you need to set it up such that it "thinks out loud" as it answers the question, generating intermediate queries against your data sources in the process.

Take a look at this chat log from Bing. Note that the user only sees the stuff that is marked #message and #suggestions.

https://www.make-safe-ai.com/is-bing-chat-safe/Prompts_Conve...

Then take a look at https://langchain.readthedocs.io/en/latest/, specifically the Document Loaders and the Indexes sections to begin with.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by a_bonobo on Tuesday March 14, 2023 @03:07AM (#35148253)
I've been playing a lot with the langchain Document Loaders - their examples make the results look more impressive than they are, or perhaps that's because our documents are 'special' (heavy in taxonomy/biology). I've been priming it with 'if you don't know, say that you don't know, don't invent anything' and it usually ends up saying 'I don't know' to questions it can clearly answer. But yeah, working on that part!
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by Havoc on Tuesday March 14, 2023 @05:33AM (#35149092)
The licensing still seems like a significant hurdle for commercial
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by zitterbewegung on Tuesday March 14, 2023 @11:26AM (#35153022)
Nothing much unsaid but huggingface has been doing this for smaller models. What is happening is they are distilling Davinci by using llama and it’s been done before .

But this performance will still page behind OpenAI for a bit but within a month or a few months we should have something almost comparable it’s just frustrating that Stanford is blocking the code and model release when it’s just a matter of time for a full copy to come out.