News for hackers. Stuff that matters.
Re:GPU Embedding with GGML by refulgentis on Monday October 16, 2023 @12:00PM (#37901764)
Interesting...I delivered a cross-platform library including MiniLM V2 last week, I'm seeing 77 embeddings/sec versus author's 20, but I'm also running on the CPU instead of GPU and M2 Max in MacBook Pro with 64 GB of RAM vs. what I think is M2 in Mac Mini with 8 GB RAM. (benchmarks: https://github.com/Telosnex/fonnx#benchmarks)

I really appreciate what GGML does but I have this irrational feeling that it'd be...unpredictable / not suited for production IMHO. (see comments here https://news.ycombinator.com/item?id=37900083 and here https://news.ycombinator.com/item?id=37898979)

I should reevaluate, this is some deep work and I assume you didn't see anything that scared you off, other than being locked to Macs for now.

There are no comments on this comment yet.