News for hackers. Stuff that matters.

Ask HN: Devs using LLMs, how are you keeping costs low for LLM calls locally?

My project has a multi step LLM flow using gpt-4o.

While developing new features/testing locally, the LLM flow frequently runs, and I use a bunch of tokens. My openAI bill spikes.

I've made some efforts to stub LLM responses but it adds a decent bit of complexity and work. I don't want to run a model locally with ollama because I need to output to be high quality and fast.

Curious how others are handling similar situations.

( 8 comments | 4 points | on HN )
Re:Ask HN: Devs using LLMs, how are you keeping costs low fo... by throwaway888abc on Thursday July 18, 2024 @10:25AM (#40995856)
Re:Ask HN: Devs using LLMs, how are you keeping costs low fo... by spruce_tips on Thursday July 18, 2024 @11:13AM (#40996325)
options if im not using langchain?
Re:Ask HN: Devs using LLMs, how are you keeping costs low fo... by ricktdotorg on Thursday July 18, 2024 @12:21PM (#40997173)
Re:Ask HN: Devs using LLMs, how are you keeping costs low fo... by spruce_tips on Thursday July 18, 2024 @01:54PM (#40998120)
options if im not using cloudflare?
Re:Ask HN: Devs using LLMs, how are you keeping costs low fo... by throwaway888abc on Thursday July 18, 2024 @04:23PM (#40999364)
You can do it easily with object caching / function memorization patterns in any modern language which should fit your desired solution. Best of luck!
Re:Ask HN: Devs using LLMs, how are you keeping costs low fo... by retrovrv on Thursday July 18, 2024 @11:35PM (#41001850)
there's an open source ai gateway - https://github.com/Portkey-AI/gateway
Re:Ask HN: Devs using LLMs, how are you keeping costs low fo... by roh26it on Thursday July 18, 2024 @09:37PM (#41001403)
Here's a mega guide on keeping costs low with LLMs - https://portkey.ai/blog/implementing-frugalgpt-smarter-llm-u...

tl;dr: - Keep prompts short, combine prompts or make more detailed prompts but go to a smaller model - Simple and semantic cache lookups - Classify tasks and route to the best LLM using an AI gateway

Portkey.ai could help with a lot of this

Re:Ask HN: Devs using LLMs, how are you keeping costs low fo... by retrovrv on Thursday July 18, 2024 @11:36PM (#41001856)
came across this guide earlier - valuable insights. thanks for sharing!