News for hackers. Stuff that matters.

A comment on Show HN: Litellm – simple library to standardize OpenAI, Cohere, Azure LLM I/O. Parent comment

Re:Show HN: Litellm – simple library to standardize OpenAI, ... by d4rkp4ttern on Saturday August 05, 2023 @11:36PM (#37018764)
Could you elaborate on this — “wouldn’t want to cache around the endpoint instead of GPT call”. Just want to see if I’m missing an important consideration here
[ Parent | on HN ]
Re:Show HN: Litellm – simple library to standardize OpenAI, ... by detente18 on Saturday August 12, 2023 @05:48PM (#37104544)
You're probably calling openai from a backend server. If you're making the call because a user asked a question -> your endpoint received it -> did some processing -> called openai -> returned the response,

why would you cache the openai call instead of the endpoint that's receiving the user call?

[ Parent | on HN ]