News for hackers. Stuff that matters.

A comment on Ask HN: Devs using LLMs, how are you keeping costs low for LLM calls locally?.

Re:Ask HN: Devs using LLMs, how are you keeping costs low fo... by roh26it on Thursday July 18, 2024 @09:37PM (#41001403)
Here's a mega guide on keeping costs low with LLMs - https://portkey.ai/blog/implementing-frugalgpt-smarter-llm-u...

tl;dr: - Keep prompts short, combine prompts or make more detailed prompts but go to a smaller model - Simple and semantic cache lookups - Classify tasks and route to the best LLM using an AI gateway

Portkey.ai could help with a lot of this

[ Parent | on HN ]
Re:Ask HN: Devs using LLMs, how are you keeping costs low fo... by retrovrv on Thursday July 18, 2024 @11:36PM (#41001856)
came across this guide earlier - valuable insights. thanks for sharing!
[ Parent | on HN ]