Re:Gemini Flash
by mcbuilder on Tuesday May 14, 2024 @03:47PM
(#40359349)
That per exchange context cost is what really puts me off using cloud LLM for anything serious. I know batching and everything is needed in the data center, and important for keeping around KVQ cache, you basically need to fully take over machine to get an interactive session to get the context costs to scale with sequence length. So it's useful, but more in the case of a local LLaMA type situation if you want a conversation.
[ Parent
| on HN ]
I wonder if we could implement the equivalent of a JIT compilation, whereby context sequences which get repeatedly reused would be used for an online fine-tuning.
Re:Gemini Flash
by HumanOstrich on Tuesday May 14, 2024 @10:10PM
(#40362360)
This is like some developer terms mixed into a word salad that might sound good to an amateur but makes no sense in the current context.
Re:Gemini Flash
by nostrebored on Tuesday May 14, 2024 @10:49PM
(#40362586)
They are asking if you can take the context being passed per interaction and train it into a session in real time (via an online algorithm).
Essentially bake the context passed in to the attention layer so that you can pass only the relevant chat context.
Your post wasn’t a particularly charitable interpretation.
No, but you can just cache the state after processing the prompt. https://github.com/ggerganov/llama.cpp/tree/master/examples/...
It makes building any app that requires generous user prompting impossible to build for regular developers (cloud pricing).
$20 hosting can serve thousands of users per month. $20 llm sub services just one person. This is fucking impossible.
Re:Gemini Flash
by cyrusalwati on Sunday May 19, 2024 @08:07PM
(#40410728)
How are you hosting for $20 per month? I've been under the impression that I'd need to pay closer to $200 to self host.
I'm building a small assistant tool and thought I'm forced to use APIs!