News for hackers. Stuff that matters.
Re:Show HN: You don't need to adopt new tools for LLM observ... by tomgs on Wednesday February 14, 2024 @11:45AM (#39371958)
Cool! Two questions:

1. Where do you see this observability for LLM thing going? What's the end game? Is it like in traditional observability where all formats eventually will converge to one format (which OpenTelemetry is trying to be)? I feel it might be a little bit early to tell, tho

2. I noticed you do auto-detection of the framework used, like LLamaIndex et al. Except for annotations, is there a deeper connection to the LLM framework used? This is auto-instrumentation, so I assume you do most of the heavy lifting, but should users of this framework expect some cool hidden eggs when they look at their telemetry?

Re:Show HN: You don't need to adopt new tools for LLM observ... by tomerf2 on Wednesday February 14, 2024 @11:48AM (#39372002)
Thanks!

1. Huh, good question. Hopefully there will be convergence. We started discussing with other companies in this domain to support or even switch to OpenTelemetry.

2. Nothing specific, except for - as you mentioned - being able to see trace of a RAG pipeline automatically.

Re:Show HN: You don't need to adopt new tools for LLM observ... by tomgs on Wednesday February 14, 2024 @11:54AM (#39372073)
A quick one :)

While we're on the topic - how does traceloop factor into all of this? What's the connection between the two? I assume the former is the LLM observability platform (Datadog for LLM?) and the latter is your own auto-instrumentation thingie to supplement it?

Re:Show HN: You don't need to adopt new tools for LLM observ... by nirga on Wednesday February 14, 2024 @11:58AM (#39372130)
Yes, Traceloop is kind of a Sentry for LLMs
Re:Show HN: You don't need to adopt new tools for LLM observ... by ssijak on Wednesday February 14, 2024 @02:31PM (#39374071)
Its priced way high, 500$ for 50k LLM calls? 50k is not much at all.
Re:Show HN: You don't need to adopt new tools for LLM observ... by nirga on Wednesday February 14, 2024 @03:34PM (#39374937)
The open source is free for all ofc. Our platform provides capabilities for monitoring and detecting hallucinations hence cost more.