News for hackers. Stuff that matters.

A comment on Show HN: Privy - Open-source, privacy oriented alternative to GitHub Copilot.

Re:Show HN: Privy - Open-source, privacy oriented alternativ... by ignoramous on Wednesday January 17, 2024 @03:26PM (#39032970)
Nice.

> Some of the popular LLMs that we recommend are: Mistral, CodeLLama

1. Surprised Mistral (Mixtral?) is recommended for code generation / explanation alongside a fine-tuned CodeLlama?

2. Recent human evals put Microsoft's WaveCoder-Ultra-6.7B (SoTA w/ GPT4) at the top of the rankings with WizardCoder-33B, Magiccoder-S-DS-6.7B trailing: https://twitter.com/TeamCodeLLM_AI/status/174755128687745064...

[ Parent | on HN ]
Re:Show HN: Privy - Open-source, privacy oriented alternativ... by ParetoOptimal on Wednesday January 17, 2024 @05:13PM (#39034299)
Typically when a 6.7B model or similar beats a 33B model it's not really true in my experience. At the least I have very a high burden of proof before believing it.
[ Parent | on HN ]
Re:Show HN: Privy - Open-source, privacy oriented alternativ... by austhrow743 on Wednesday January 17, 2024 @05:32PM (#39034520)
Are you able explain what the charts mean? Only one of the three has wavecoder at the top.
[ Parent | on HN ]
Re:Show HN: Privy - Open-source, privacy oriented alternativ... by ignoramous on Wednesday January 17, 2024 @07:19PM (#39035619)
Those charts show pass@k metric (expectation at least k generated samples are correct out of n) on OpenAI and Octopack problem evals for code.

WaveCoder: https://arxiv.org/abs/2312.14187 (section 3.2)

Octopack: https://github.com/bigcode-project/octopack

[ Parent | on HN ]
Re:Show HN: Privy - Open-source, privacy oriented alternativ... by srikanth235 on Wednesday January 17, 2024 @07:52PM (#39035968)
While testing internally, Mistral worked well. But these models are just starting points. Will add support for the models WaveCoder-Ultra-6.7B, WizardCoder-33B, Magiccoder-S-DS-6.7B, etc soon.
[ Parent | on HN ]