Re:I Run LLMs Locally
by adam_arthur on Sunday December 29, 2024 @04:37PM
(#42543479)
Whether performance or cost is more important depends on your use case. Some tasks that an LLM can do very well may not need to be done often, or even particularly quickly (as in my case).
e.g. LLM as one step of an ETL-style pipeline
Latency of the response really only matters if that response is user facing and is being actively awaited by the user
e.g. LLM as one step of an ETL-style pipeline
Latency of the response really only matters if that response is user facing and is being actively awaited by the user