News for hackers. Stuff that matters.
Re:Apple: Transformer architecture optimized for Apple Silicon by johnthuss on Thursday March 23, 2023 @11:25PM (#35284714)
What prompt are you using for summarization? I’ve tried several variations without consistent results.
Re:Apple: Transformer architecture optimized for Apple Silicon by garblegarble on Friday March 24, 2023 @09:07AM (#35288521)
On alpaca, I've found "Below is an instruction that describes a task. Write a response that appropriately completes the request. Summarise the following text: " or "Give me a 5 word summary of the following: " to work fairly well using the 30B weights.

It's certainly nowhere close to the quality of OpenAI summarisation, just better than what I previously had locally (e.g. in summarising a family history project with transcripts of old letters, gpt-3.5-turbo was able to accurately read between the lines summarising an original poem which I found amazing).

I half wonder if the change in spelling from US -> UK makes a difference...

I'd run a test on that but I've just broken my alpaca setup for longer prompts (switched to use mainline llama.cpp, which required a model conversion & some code changes, and it's no longer allocating enough memory)