News for hackers. Stuff that matters.

A comment on Show HN: Privy - Open-source, privacy oriented alternative to GitHub Copilot. Parent comment

Re:Show HN: Privy - Open-source, privacy oriented alternativ... by austhrow743 on Wednesday January 17, 2024 @05:32PM (#39034520)
Are you able explain what the charts mean? Only one of the three has wavecoder at the top.
[ Parent | on HN ]
Re:Show HN: Privy - Open-source, privacy oriented alternativ... by ignoramous on Wednesday January 17, 2024 @07:19PM (#39035619)
Those charts show pass@k metric (expectation at least k generated samples are correct out of n) on OpenAI and Octopack problem evals for code.

WaveCoder: https://arxiv.org/abs/2312.14187 (section 3.2)

Octopack: https://github.com/bigcode-project/octopack

[ Parent | on HN ]
Re:Show HN: Privy - Open-source, privacy oriented alternativ... by srikanth235 on Wednesday January 17, 2024 @07:52PM (#39035968)
While testing internally, Mistral worked well. But these models are just starting points. Will add support for the models WaveCoder-Ultra-6.7B, WizardCoder-33B, Magiccoder-S-DS-6.7B, etc soon.
[ Parent | on HN ]