Whisper Jax: Transcribe a 1 hour of audio in under 15 seconds

Re:Whisper Jax: Transcribe a 1 hour of audio in under 15 sec... by sebnun on Saturday April 22, 2023 @06:13AM (#35664988)

I've been looking for faster implementations of Whisper, the main drawback with Whisper Jax is that the performance comes from running on Google TPUs, which are much more expensive than GPUs.

On "normal" GPUs the fastest implementation I've found is https://github.com/guillaumekln/faster-whisper. Whisper.cpp works faster on a CPU, especially on Apple Silicon, but still nowhere near the performance you could get on a GPU (understandably).

How does Whisper Jax compares to faster-whisper on a GPU?

[ Parent | on HN ]

Re:Whisper Jax: Transcribe a 1 hour of audio in under 15 sec... by ttt3ts on Saturday April 22, 2023 @09:32AM (#35666443)

Yea, whisper runs fairly quickly on really old GPUs and leveraging CTranslate2 (underlying tech in your link) you need < 4GB of GPU RAM to run the large model.

This might be faster but an order of magnitude more expensive? Also, huge lockin to one provider.

[ Parent | on HN ]

Re:Whisper Jax: Transcribe a 1 hour of audio in under 15 sec... by zznzz on Saturday April 22, 2023 @12:19PM (#35668088)

jax and whisper-jax support running on GPUs in addition to TPUs, this particular demo just runs on a TPU.

[ Parent | on HN ]