Local LLM inference – impressive but too hard to work with

Re:Local LLM inference – impressive but too hard to work with by aazo11 on Monday April 21, 2025 @04:01PM (#43755847)

Exactly. Why does this not exist yet?

[ Parent | on HN ]

Re:Local LLM inference – impressive but too hard to work with by byyoung3 on Monday April 21, 2025 @04:33PM (#43756153)

its an if statement on whether the model has downloaded or not

[ Parent | on HN ]

Re:Local LLM inference – impressive but too hard to work with by aazo11 on Monday April 21, 2025 @05:20PM (#43756622)

A better solution would train/finetune the smaller model from the responses of the larger model and only push to the inference to the edge if the smaller model is performant and the hardware specs can handle the workload?

[ Parent | on HN ]

Re:Local LLM inference – impressive but too hard to work with by monoid73 on Monday April 21, 2025 @06:21PM (#43757145)

yeah, that'd b nice, some kind of self-bootstrapping system where you start with a strong cloud model, then fine-tune a smaller local one over time until it’s good enough to take over. tricky part is managing quality drift and deciding when it's 'good enough' without tanking UX. edge hardware's catching up though, so feels more feasible by the day.

[ Parent | on HN ]