News for hackers. Stuff that matters.
Re:Show HN: Any-LLM – Lightweight router to access any LLM P... by swyx on Tuesday July 22, 2025 @03:13PM (#44651721)
> LiteLLM: While popular, it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications

with no vested interest in litellm, i'll challenge you on this one. what compatibility issues have come up? (i expect text to have the least, and probably voice etc have more but for text i've had no issues)

you -want- to reimplement interfaces because you have to normalize api's. in fact without looking at any-llm code deeply i quesiton how you do ANY router without reimplementing interfaces. that's basically the whole job of the router.

Re:Show HN: Any-LLM – Lightweight router to access any LLM P... by Szpadel on Tuesday July 22, 2025 @03:48PM (#44652157)
I use litellm as my personal AI gateway, and from user point of view there is no difference if proxy uses official SDK or not, this might be benefit for proxy developers.

but I can give you one example: litellm recently had issue with handling deepseek reasoning. they broke implementation and while reasoning was missing from sync and streaming responses.

Re:Show HN: Any-LLM – Lightweight router to access any LLM P... by delijati on Tuesday July 22, 2025 @03:40PM (#44652061)
there is nothing lite in litellm ... i was experimenting (using as a lib) but ended using https://llm.datasette.io/en/stable/index.html btw. thanks @simonw for llm
Re:Show HN: Any-LLM – Lightweight router to access any LLM P... by scosman on Tuesday July 22, 2025 @03:36PM (#44652025)
Yeah, official SDKs are sometimes a problem too. Together's included Apache Arrow, a ~60MB dependency, for a single feature (I patched to make it optional). If they ever lock dependency versions it could conflict with your project.

I'd rather a library that just used OpenAPI/REST, than one that takes a ton of dependencies.

Re:Show HN: Any-LLM – Lightweight router to access any LLM P... by chuckhend on Tuesday July 22, 2025 @03:19PM (#44651811)
LiteLLM is quite battle tested at this point as well.

> it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications

Leveraging official SDKs also does not solve compatibility issues. any_llm would still need to maintain compatibility with those offical SDKs. I don't think one way clearly better than the other here.

Re:Show HN: Any-LLM – Lightweight router to access any LLM P... by AMeckes on Tuesday July 22, 2025 @03:40PM (#44652059)
That's true. We traded API compatibility work for SDK compatibility work. Our bet is that providers are better at maintaining their own SDKs than we are at reimplementing their APIs. SDKs break less often and more predictably than APIs, plus we get provider-implemented features (retries, auth refresh, etc) "for free." Not zero maintenance, but definitely less. We use this in production at Mozilla.ai, so it'll stay actively maintained.
Re:Show HN: Any-LLM – Lightweight router to access any LLM P... by amanda99 on Tuesday July 22, 2025 @03:24PM (#44651882)
Being battle tested is the only good thing I can say about LiteLLM.
Re:Show HN: Any-LLM – Lightweight router to access any LLM P... by scosman on Tuesday July 22, 2025 @03:31PM (#44651953)
You can add in it's still 10x better than LangChain
Re:Show HN: Any-LLM – Lightweight router to access any LLM P... by AMeckes on Tuesday July 22, 2025 @04:19PM (#44652518)
Both approaches work well for standard text completion. Issues tend to be around edge cases like streaming behavior, timeout handling, or new features rolling out.

You're absolutely right that any router reimplements interfaces for normalization. The difference is what layer we reimplement at. We use SDKs where available for HTTP/auth/retries and reimplement normalization.

Bottom line is we both reimplement interfaces, just at different layers. Our bet on SDKs is mostly about maintenance preferences, not some fundamental flaw in LiteLLM's approach.