Show HN: OpenLIT – Open-Source LLM Observability with OpenTelemetry

Re:Show HN: OpenLIT – Open-Source LLM Observability with Ope... by patcher99 on Sunday April 28, 2024 @11:04PM (#40193991)

Hey! On the metrics front, we do basic mettics like requests, tokens and cost for now as all of the usefull information about llm is included directly in the Span Attributes of the Trace.

So what a lot of users have told me is the want to track their usage(cost/tokens) and want to keep an eye on the user interactions like prompts and responses, We are looking to add more accuracy based metrics too which is very highly asked to us.

Also re on UI: UI is optional, You can use the sdk to directly send otel traces and meyrics to your preferred destination. https://docs.openlit.io/latest/connections/intro

[ Parent | on HN ]

Re:Show HN: OpenLIT – Open-Source LLM Observability with Ope... by arwineap on Monday April 29, 2024 @09:42AM (#40198236)

On the span is an interesting attribute to view, but had you considered using otel meters/metrics for that instead?

I think cardinality in span attributes can be a problem, and meters are better for aggregating and graphing

[ Parent | on HN ]

Re:Show HN: OpenLIT – Open-Source LLM Observability with Ope... by patcher99 on Monday April 29, 2024 @10:06AM (#40198515)

We do Otel metrics for the main things needed in dashboarding 1. Requests (Counter) 2. Tokens (Counter) (Seperate metric for Prompt Tokens, Completion and Total Tokens) 3. Cost (Histogram)

I did attach a Grafana dashboard too (Works for Grafana Cloud, Ill get something for OSS this week) https://docs.openlit.io/latest/connections/grafanacloud

Since Otel doesn't yet support Synchronous gauge + Users wanted a trace of the LLM RAG application, We opted to Traces which is quite standard now in LLM Observability tool.

Lemme know if you had something more in mind, Love getting feedback from amazing folks like you!

[ Parent | on HN ]