News for hackers. Stuff that matters.

A comment on Show HN: OpenLIT – Open-Source LLM Observability with OpenTelemetry. Parent comment

Re:Show HN: OpenLIT – Open-Source LLM Observability with Ope... by arwineap on Monday April 29, 2024 @09:42AM (#40198236)
On the span is an interesting attribute to view, but had you considered using otel meters/metrics for that instead?

I think cardinality in span attributes can be a problem, and meters are better for aggregating and graphing

[ Parent | on HN ]
Re:Show HN: OpenLIT – Open-Source LLM Observability with Ope... by patcher99 on Monday April 29, 2024 @10:06AM (#40198515)
We do Otel metrics for the main things needed in dashboarding 1. Requests (Counter) 2. Tokens (Counter) (Seperate metric for Prompt Tokens, Completion and Total Tokens) 3. Cost (Histogram)

I did attach a Grafana dashboard too (Works for Grafana Cloud, Ill get something for OSS this week) https://docs.openlit.io/latest/connections/grafanacloud

Since Otel doesn't yet support Synchronous gauge + Users wanted a trace of the LLM RAG application, We opted to Traces which is quite standard now in LLM Observability tool.

Lemme know if you had something more in mind, Love getting feedback from amazing folks like you!

[ Parent | on HN ]