News for hackers. Stuff that matters.

A comment on We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation. Parent comment

Re:We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for ... by siscia on Thursday January 02, 2025 @08:15AM (#42574124)
DeepSeek is an amazing product but has few issues:

1. Data is used for training

2. Context window is rather small and doesn't fit as well large codebase

I keep saying this over and over in all the content I create, the valu of coding with AI will come from working on big, complex, legacy codebases. Not from flashy demo where you create a to-do app.

For that you need solid models with big context and private inference.

[ Parent | on HN ]
Re:We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for ... by MacsHeadroom on Thursday January 02, 2025 @08:20AM (#42574161)
DeepSeek is open source and has a context length of 128k tokens.
[ Parent | on HN ]
Re:We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for ... by siscia on Thursday January 02, 2025 @08:30AM (#42574224)
Commercial service have a context of 64k tokens, which I find quite limiting.

https://api-docs.deepseek.com/quick_start/pricing

Running it locally is quite a bit beyond the scope of being productive while coding with AI.

Beside that 128k is still significantly less than Claude

[ Parent | on HN ]
Re:We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for ... by elashri on Thursday January 02, 2025 @08:52AM (#42574386)
Shouldn't we be comparing with other open source model? In particular since this is about llama3.3 then they have the exact context limit which is 128k [1]. Also

[1] https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct

[ Parent | on HN ]
Re:We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for ... by siscia on Thursday January 02, 2025 @09:18AM (#42574607)
Why?

Whenever using a model to be more effective as a developer I don't particularly care if the model is open source or closed source.

I would love to use open source models as well, but the convenience to just plug an API against some endpoints in unbeatable.

[ Parent | on HN ]