News for hackers. Stuff that matters.
Re:Apple: Transformer architecture optimized for Apple Silicon by qumpis on Thursday March 23, 2023 @09:03PM (#35283719)
Offtopic, but for what purpose are you running llms locally (especially everyday)? My understanding was that the prompting requires to make them work at all was too great.
Re:Apple: Transformer architecture optimized for Apple Silicon by garblegarble on Thursday March 23, 2023 @09:29PM (#35283919)
A little bit of research, a little bit of actual useful tasks - I'm interested in summarisation, which alpaca is decent at (even compared to existing summarisation-specific models I've tried)

My other motivation is making sure I understand what offline LLMs can do... while I use GPT-3 and 4 extensively, I don't want to send something over the wire if I don't have to (e.g. if I can summarise e-mails locally, I'd rather do that than send them to OpenAI).

It's also surprisingly good at defining things if I'm somewhere with no internet connectivity and want to look something up (although obviously that's not really what it's good at & hallucination risks abound)

Re:Apple: Transformer architecture optimized for Apple Silicon by johnthuss on Thursday March 23, 2023 @11:25PM (#35284714)
What prompt are you using for summarization? I’ve tried several variations without consistent results.
Re:Apple: Transformer architecture optimized for Apple Silicon by garblegarble on Friday March 24, 2023 @09:07AM (#35288521)
On alpaca, I've found "Below is an instruction that describes a task. Write a response that appropriately completes the request. Summarise the following text: " or "Give me a 5 word summary of the following: " to work fairly well using the 30B weights.

It's certainly nowhere close to the quality of OpenAI summarisation, just better than what I previously had locally (e.g. in summarising a family history project with transcripts of old letters, gpt-3.5-turbo was able to accurately read between the lines summarising an original poem which I found amazing).

I half wonder if the change in spelling from US -> UK makes a difference...

I'd run a test on that but I've just broken my alpaca setup for longer prompts (switched to use mainline llama.cpp, which required a model conversion & some code changes, and it's no longer allocating enough memory)

Re:Apple: Transformer architecture optimized for Apple Silicon by bigfudge on Friday March 24, 2023 @04:38AM (#35286529)
Necessary if you have sensitive datasets you can’t share with US company