Llama.cpp: Full CUDA GPU Acceleration

Re:Llama.cpp: Full CUDA GPU Acceleration by aidenn0 on Tuesday June 13, 2023 @12:53AM (#36305531)

Slightly OT:

I have been playing around with whisper.cpp; it's nice because I can run the large model (quantized to 8-bits) at roughly real-time with cublas on a Ryzen 2700 with a 1050Ti. I couldn't even run the pytorch whisper medium on this card with X11 also running.

It blows me away that I can get real-time speech-to-text of this quality on a machine that is almost 5 years old.

[ Parent | on HN ]

Re:Llama.cpp: Full CUDA GPU Acceleration by moneywoes on Tuesday June 13, 2023 @01:02AM (#36305622)

Is it possible to run on apple m1 devices or mobile phones or not yet?

[ Parent | on HN ]

Re:Llama.cpp: Full CUDA GPU Acceleration by raihansaputra on Tuesday June 13, 2023 @01:08AM (#36305679)

yeah the whisper.cpp github page has a demo for both. Have used it on my M1 MBA for the past few months.

[ Parent | on HN ]

Re:Llama.cpp: Full CUDA GPU Acceleration by michelb on Tuesday June 13, 2023 @02:01AM (#36306099)

I can recommend the MacWhisper app if you prefer a gui.

[ Parent | on HN ]

Re:Llama.cpp: Full CUDA GPU Acceleration by Void_ on Tuesday June 13, 2023 @03:31AM (#36306810)

And Whisper Memos for iOS https://whispermemos.com/

[ Parent | on HN ]

Re:Llama.cpp: Full CUDA GPU Acceleration by b33f on Tuesday June 13, 2023 @03:46AM (#36306934)

The really nice part of Whisper is being able to use it offline and on-device, it seems whisper memos is uploading your audio and notes to a server of unknown security, confidentiality etc.

I like Aiko for on-device transcription both in macOS and iOS https://apps.apple.com/us/app/aiko/id1672085276

[ Parent | on HN ]

Re:Llama.cpp: Full CUDA GPU Acceleration by Void_ on Tuesday June 13, 2023 @04:42AM (#36307392)

Whisper Memos uses OpenAI API. The upside is that it uses the largest model - that would take 2GB on your iPhone.

[ Parent | on HN ]

Re:Llama.cpp: Full CUDA GPU Acceleration by nitinreddy88 on Tuesday June 13, 2023 @02:19AM (#36306236)

is there any dummy guide to get started with any of these?

[ Parent | on HN ]

Re:Llama.cpp: Full CUDA GPU Acceleration by Tepix on Tuesday June 13, 2023 @04:32AM (#36307316)

Have you tried the Quick start in the https://github.com/ggerganov/whisper.cpp README?

[ Parent | on HN ]

Re:Llama.cpp: Full CUDA GPU Acceleration by eurekin on Tuesday June 13, 2023 @08:08AM (#36309043)

Seconded. I were playing around for my native language (Polish) and the large models actually blew me away. For example, it handled "przescreenować" spelling correctly, which is an english word with a polish prefix and a conjugated suffix.

[ Parent | on HN ]

Re:Llama.cpp: Full CUDA GPU Acceleration by m3affan on Tuesday June 13, 2023 @08:14AM (#36309098)

This is an impresdive use case

[ Parent | on HN ]