If you don't run your own LLM proxy, how do you even know whats going on?

Re:If you don't run your own LLM proxy, how do you even know... by hoppp on Sunday October 18, 2026 @08:46PM (#50147059)

Yup. Setting up liteLLMProxy on some servers as we speak and will start playing around.

I can't orchestrate agents in sandboxes without it.

1. It can issue virtual keys, so provider API keys never go near the sandbox

2. I can see everything

3. Guardrails

And much more

Looking forward to it.

[ Parent | on HN ]

Re:If you don't run your own LLM proxy, how do you even know... by DelightOne on Sunday October 18, 2026 @09:07PM (#50147181)

1) What kind of guardrails are you using? Do you do models like prompt injection checkers? Or what guardrails are useful that do not know the environment the agent lives in?

2) I heard Bifrost also works for this. Did you try that as well, pros/cons?

[ Parent | on HN ]

Re:If you don't run your own LLM proxy, how do you even know... by dcss_gardener on Sunday October 18, 2026 @09:51PM (#50147437)

I'm not the person you asked but yes pretty much. I've found the prompt injection protection models pretty limited, but get good results from gpt-oss-safeguard with a policy describing various prompt injection techniques. More expensive but more flexible and reliable and I only run it on web fetch and web search results. Costs me like 5-20 cents a day depending on what I'm working on.

[ Parent | on HN ]

Re:If you don't run your own LLM proxy, how do you even know... by hoppp on Monday October 19, 2026 @10:58AM (#50153297)

Still working on deploying it and setting it up with cloudflare sandboxes so I don't have deep expertise and need to experiment first before I can give advice

[ Parent | on HN ]

Re:If you don't run your own LLM proxy, how do you even know... by ij23 on Monday October 19, 2026 @01:21PM (#50155176)

Hi i'm a LiteLLM maintainer thanks for using us, I'd love to connect to hear how we can improve for you. Here's my Linkedin https://www.linkedin.com/in/reffajnaahsi/ / Twitter (https://x.com/ishaan_jaff) if you're open to a convo

[ Parent | on HN ]

Re:If you don't run your own LLM proxy, how do you even know... by hoppp on Monday October 19, 2026 @07:24PM (#50159358)

As feedback I would say the UI should be more mobile friendly.

I need to use the desktop site checkbox on chrome in android otherwise the UI is mushed together.

[ Parent | on HN ]