Hi HN, Spike and Donovan here. We have been building BricksLLM (https://github.com/bricks-cloud/bricksllm), an API gateway that allows you to add the "OpenAI DX" to any arbitrary LLMs. For example, you can create a self-serve dashboard to allow end-users to create API keys for accessing your LLMs, impose different rate/spend limits for each end-user, track their token usage, and detect PII....
* Litellm is declarative and it let you define everything in yaml * Bricks is not declarative and you control everything via API
* Litellm does not have an UI * Bricks has a non open source UI
* Litellm is written in python * Bricks is written in Golang
* Litellm does not persist rate limits. Therefore can't accurately rate limit across distributed instances * Bricksllm let you create API keys with accurate rate limits and spend limits that work across distributed instances
* Litellm provides high level spend metrics on API keys * Bricks provides granular spend, request and latency metrics breakdown by model and custom id
* Litellm is not compatible with OpenAI SDK. You have to adopt Litellm python client * Bricks is designed to be compatible with OpenAI SDK
* Litellm only supports OpenAI completion and embedding * Bricks supports almost all OpenAI endpoints except image and audio
* Litellm has exact request caching * Bricks does not have caching as for now
* Litellm has OpenTelemetry integration * Bricks has statsd integration
* Litellm supports orchestration of API calls. When this API call fails, use this model or call this API endpoint instead * Bricks does not support orchestration of API calls since I believe that it is something that the client should handle
https://github.com/bricks-cloud/BricksLLM
Hey HN, I am proud to show you guys that I have built an open source alternative to Azure OpenAI services.Azure OpenAI services was born out of companies needing enhanced security and access control for using different GPT models. I want to build an OSS version of Azure OpenAI services that people could self host in their own infrastructure."How can I track LLM spend per API key?""Can I create...
We developed an open source plugin that let you go from Figma to tailwindcss + react. Feel free to check it out
https://www.figma.com/community/plugin/1178847414663679049/B...