Show HN: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

Re:Show HN: Beating GPT-4 on HumanEval with a fine-tuned Cod... by sodality2 on Saturday August 26, 2023 @02:04PM (#37275259)

From my experience llama.cpp doesn’t take full advantage of parallelism as it could. Tested this on an HPC cluster - increasing thread count certainly did increase CPU usage but did not meaningfully improve tok/s past 6-8 cores. Same behavior with whisper.cpp. :( I wonder if there’s another backend that scales better.

[ Parent | on HN ]

Re:Show HN: Beating GPT-4 on HumanEval with a fine-tuned Cod... by moonchrome on Saturday August 26, 2023 @03:53PM (#37276247)

I'm guessing the problem is that you're constrained by memory bandwidth and not computation power and this is inherent to the algorithm, not an artifact of any one implementation.

[ Parent | on HN ]