Weves

Have you checked out https://docs.onyx.app/deployment/getting_started/quickstart#...?

We should definitely update the Local guide to reflect the ^, thanks for pointing that out.

For the container footprint - pretty much all of them are necessary to run locally. We hope that just a large # of containers shouldn't be a problem, as long as they are all lightweight (e.g. total resource utilization is low). We of course could bundle them together into a single "mega container", but that seems against the best practices of Docker.

On a related note, we are looking to have a version that doesn't require the vector database (the heaviest part of the system in terms of resource utilization). Our goal is to have a deployment mode with less than 1gb of RAM required.

Yep, we have our own implementations! We've spent a lot of time on them, and in our internal benchmarks they compare pretty favorable to the native versions.

RAG specifically is our speciality - we've done a ton of optimizations there (hybrid search, document age-based weighting, giving the LLM the ability to read more from interesting docs and less of irrelevant docs, etc.) and we outperform the implementation within ChatGPT quite substantially in internal blind testing.

Curious what you find if you compare them head to head though!

A few reasons:

1/ No model lock-in / ability to use the ideal model for each use case.

2/ More connectivity. A fuller connector library (contributed to by the open-source community). More built-in tools (similar to the ^).

3/ Customizability and flexibility. If you really need a feature, you can build it rather than waiting months (years?) for your request to go through.

4/ White-labeling. You can make it feel like a product built for/by your company rather than generic.

Reposting from https://news.ycombinator.com/item?id=46047430:

---

Broadly, I think other open source solutions are lacking in (1) integration of external knowledge into the chat (2) simple UX (3) complex "agent" flows. Both internal RAG and web search are hard to do well, and since we've started as an enterprise search project we've spent a lot of time making it good.

Most (all?) of these projects have UXs that are quite complicated (e.g. exposing front-and-center every model param like Top P without any explanation, no clear distinction between admin/regular user features, etc.). For broader deployments this can overwhelm people who are new to AI tools.

Finally trying to do anything beyond a simple back and forth with a single tool calls isn't great with a lot of these projects. So something like "find me all the open source chat options, understand their strengths/weaknesses, and compile that into a spreadsheet" will work well with Onyx, but not so well with other options (again partially due to our enterprise search roots).

You could! We have a number of users who do something like that (e.g. Ramp's in-app chat bot is powered by Onyx)

Great suggestion! I haven't heard of it before, but I'll definitely add it to the list.

Would love to hear your feedback after you get a chance to try Onyx out.

Thanks! Especially these days, refinement is (by far) the hardest part :)

> One thing I really care about is extensibility

Totally makes sense! We've defined simple `Connector`, `Tool` and (soon) `Agent` interfaces to make it easy to plug in your own implementations/apps. If you wanted, you could just use Langchain under our Agent class to build arbitrary flows.

Additionally, the main chat is created from a series of `Renderer` components, and it should be easy to build your own.

Do you think that's in-line with what you're thinking of, or do you want to build outside of those confines?

> canvas mode and scheduled actions

Yes, writing and recurring are two big areas we want to tackle next year.

> The main thing I really care about is voice mode

Interesting. Why do you think that is?

We have! Likely not for a while, since we want to focus on the web app for a bit more. But long term that would be the plan.

Things like shortcuts to access the chat + ability to reach into the local file system are very compelling.

Yes indeed! Our belief is that tool design, compaction (e.g. tool result summarization), and reminders are the things that separate a product that works magically vs one that falls over on any slightly more complex task.

And this is all made all the more important when supporting the wide range of models, even "weaker" open-source models.

> ease of installation, streaming support, model agnosticity, chat persistence and blob support

we have all of those!

> how onyx is comparable

For an AI-powered research assistant, Onyx might just work out of the box. We have ~45 connectors to common apps (https://github.com/onyx-dot-app/onyx/blob/main/backend/onyx/...), integrations with the most popular web search providers (https://github.com/onyx-dot-app/onyx/blob/main/backend/onyx/...), and a built in tool calling loop w/ deep research support (https://github.com/onyx-dot-app/onyx/blob/main/backend/onyx/...). If you wanted to customize, you could pretty easily tweak this / add additional tools (or even rip this out completely and build your own agent loop).

What format are the docs being uploaded as? By default, images uploaded into the chat would be directly passed through. PDFs would be parsed and fed to the LLM as text.

Writing is a really common use case, and something we'd like to explore more. Currently people often use Onyx for "write something combining X, Y, and Z documents", but I feel that's just scratching the surface.

That's one way people use Onyx! Specifically, the Projects feature (see the left sidebar) works similarly e.g. you can upload arbitrary numbers of files - going well beyond the context limit of your model - and then ask questions of them.

Totally get this concern.

How we think about it: the chat product should be completely open-source and free (forever). To that end we've moved features like SSO (that used to be "enterprise") to be MIT licensed. The chat interface is something pretty much every team needs (be it a proprietary or open-source solution). You can think of this like Apache Spark for Databricks or Ray for Anyscale.

Also, as other folks have pointed out in the thread, there are quite a few other open source options out there. So there's a a ton of outside pressure for our open-source only offering to be very competitive. We hope this reduces the "enshitification" risk that you speak of.

Thanks! And 404 should be fixed

Haha, yea the UIs certainly have similarities (much of the industry converges to standard places to put different components, since users are familiar).

"Agents" is a particular area where we feel like we're better than the alternatives (especially if you want something that effectively calls multiple tools in sequence). Curious to hear your thoughts after trying it out!

Hmm, will have to disagree here. I think "one chat to rule them all" is the way it will end.

It does requires having UI components for many different types of interactions (e.g. many ways to collect user input mid-session + display different tools responses like graphs and interactives). With this, people should be able to easily build complex tools/flows on top of that UI, and get a nice, single interface (no siloed tools/swapping) for free. And having this UI be open-source make this easier.

Great question! Depends on the specific alternative, but the broad points are:

- "pure chat" experience. From our community (and personal use), we've observed that most queries don't actually involve enterprise search. They much more likely to just require the LLMs internal knowledge (or web search / code execution). Compared to all the companies you've mentioned, we've spent a lot more time refining this more common flow.

- Larger connector suite. As soon as one key source isn't connected, the trustworthiness of the system is dramatically decreased. You second guess "is the info needed to answer this question in there?" for every question. We have a community who builds out connectors for themselves, and then contribute it back for everyone to use. This allows us to cover the long-tail better than companies like Notion and Slack.

- Customizability. An open-source application is the perfect middle ground between a SaaS offering and building blocks. A SaaS option doesn't allow for any customization (we have many customers who have contributed back ux enhancements, small features like guardrails, or enhanced configurations that their users want). Building blocks demand too much domain expertise (search, frontend/UX, ...) for it to be realistic for companies to build something great.

Hmm, yea that's a great callout. Something we definitely have in our sights longer term (focus for now is to make sure that the desktop chat experience is truly amazing).

Yes, exactly! Would love to hear your feedback compared to OpenWebUI

Yes absolutely! There's no license restriction on white-labeling, so we've seen lots of companies do that.

In our opinion, it's a bit silly to build completely in house when you can take something like Onyx as the starting point and be >95% of the way there + have a tons of bells and whistles built in.

Outside of pure security, we've seen many choose Onyx for flexibility + connectedness.

One of our largest users has forked the repo and has 20+ commits back to the repo of small customizations that are important for them (and that they could never get with ChatGPT Enterprise).

Lots of companies we talk to value having the best model for the job (e.g. not being tied to ONLY OpenAI models for example).

Compared to model provider offerings, we also (thanks to open-source contributions) cover many more existing apps when it comes to connectors.

All of the core chat UX + "add-ons" is in FOSS!

In the enterprise:

- permission syncing

- UI-based white labeling

- Advanced RBAC

- Usage analytics UI

Thanks for the kind words!

The key point there is that many would do it through Azure / Bedrock + locally host the open-source models. Also, all chats / indexed data lives on-prem, and there are better guarantees around retention when using the APIs directly.

That's a fair point! We were aware of onnx, but felt it was okay since they are very different products so we felt that there wouldn't be too much confusion (people generally know which onyx/onnx they are looking for).

We're definitely looking to add back some of that flexibility / customizability. I don't think you have to sacrifice a nice, simple UI to provide what power users are looking for.

For now, the main reasons for a prosumer to use over oobabooga/sillytavern are around the base tool set we provide and the "agent loop". If you ever want to use your single chat interface to do data analysis (code interpreter), multi-step realtime research (deep research), or RAG over large scale data (hybrid search), Onyx would be a particularly good choice.

Great to hear that you've heard good things. And yea we have many large (>1k+) teams using just the open-source version (something we love to see).