honorable_judge

Yea - that's right. Its this separation of concerns that I think will help people break through the confusion of building agents.

Envoy is compatible with OTel out of the box. That's a big plus for observability. Plus Envoy is designed for high-load dataplane (in the request path worklaods) and used in every modern stack. There are several advantages on using Arch as the source of observability (traces, metrics, logs)

Arch moves the critical but crufty work around safety, observability, and routing of prompts outside business logic. Its a uniquely intelligent infrastructure primitive, engineered with purpose-built fast LLMs [3] for tasks like intent detection over multi-turn, parameter identification and extraction, triggering single/multiple function calls, and offers convenience features to auto dispatch LLM calls for summarization based on data from your APIs via system prompts configured in archgw.

For a programming language mostly out of favor, the effort to get a framework right and find the right audience to use this over existing options will be tricky. Good luck!

Its a proxy - built on Envoy. I think its fairly clear that this is a separate process. As far as I can tell, you create a config file, boot up archgw, and in the config have it point to endpoints where prompts get forwarded.

Found it interesting (for the use case) - the work to export chat and then map is exactly that: work

With all the focus on language specific frameworks - this out of process architecture choice is an interesting one. On one hand, it helps you side step the "is this functionality available on js, java, etc" question, and on the other it means its not as easy as `import archgw` in python. Good luck though, feels like an interesting project