Re:Cost Per 1M tokens Of Self Hosting Llama-3
by shostack on Friday June 14, 2024 @01:02PM
(#40682540)
How is inference latency for coding use cases on a local 3090 or 4090 compared to say, hitting the GPT-4o API?
[ Parent
| on HN ]
Re:Cost Per 1M tokens Of Self Hosting Llama-3
by whereismyacc on Friday June 14, 2024 @01:16PM
(#40682677)
I assume the characteristics would be pretty different, since your local hardware can keep the context loaded in memory, unlike APIs which I'm guessing have to re-load it for each query/generation?
Re:Cost Per 1M tokens Of Self Hosting Llama-3
by christina97 on Friday June 14, 2024 @01:41PM
(#40682956)
If you integrate with existing tooling, it won’t do this optimization. Unless of course you really go crazy with your setup.
Re:Cost Per 1M tokens Of Self Hosting Llama-3
by moffkalast on Friday June 14, 2024 @05:36PM
(#40685373)
Setting one launch flag on llama.cpp server hardly qualifies as going crazy with one's setup.