kakaly0403

Congrats on the launch. Cool to see a RAG specific tracing tool. Excited to try it out. Full disclosure, I am the cofounder and core-maintainer of Langtrace(https://github.com/Scale3-Labs/langtrace) which is also an open source tool for tracing and observing your LLM stack and our SDKs are OTEL based. Based on my experience, I think the biggest challenge right now specifically for RAG pipelines is the lack of flexibility in the current crop of tracing tools to not just visualize the entire retrieval flow across all the components of the stack - the framework calls, vectorDB retrievals, re-ranker i/o if any and the final LLM inference. But, also being able to do experiments by freezing a setup, iterate on it and measuring the performance and improving it to clearly know how the changes map to the performance end to end. This is what we think about mostly while we are building Langtrace as well.

Promptfoo is pretty good

RSS Feed - https://anchor.fm/s/fb0d9ea8/podcast/rss

Here you go: https://anchor.fm/s/fb0d9ea8/podcast/rss

There is a GenAI standard spec from OpenTelemetry for tracing LLM based applications. Currently there are 3 library implementations of this spec - Langtrace, OpenLLMetry and OpenLit. Microsoft has an implementation for .NET aswell. OpenInference, though opentelemetry compatible does not adhere to the standard spec.

The way it works: - I give it the content as a pdf or a text - it generates the script and records the audio

Langtrace core maintainer here. Congrats on the launch! We are building OTEL support for a wide range of LLMs, vectorDBs and frameworks - crewai, DSPy, langchain etc. Would love to see if the langtrace’s tracing library can be integrated with Laminar. Also, feel free to join the OTEL GenAI semantic working committee.

Checkout Langtrace. It’s also OTEL and integrates with datadog.

https://github.com/Scale3-Labs/langtrace

1. You do not need 2 databases if you are only looking to export the traces to your grafana stack. If you are using prometheus as the TSDB, you can directly send the traces there without needing any of the databases to be setup locally.

2. Also, you do not need the API key. You only need to install the python or typescript sdk and use the custom exporter option - https://github.com/Scale3-Labs/langtrace-python-sdk?tab=read... . If you already have a OTEL exporter running, you don't need even that. Just initialize the sdk after installing and it will do its thing.

I am one of the core maintainers of the project. Do let me know if you have any questions.

If you are looking for automatic instrumentation that generates open telemetry standard traces - check out Langtrace, https://github.com/Scale3-Labs/langtrace. Got support for all popular LLMs, frameworks and vectorDB with high cardinality.

Cool. Have you considered adopting to the official opentelemetry standards? https://opentelemetry.io/docs/specs/semconv/attributes-regis...

Hey I am building an open telemetry based open source tool for metrics and logs from LLM based apps. Give it a try at langtrace.ai

+1 to this. Langtrace core maintainer here. We are building a SDK and a client for automatic instrumentation of LLM based applications using OTEL standards. The OpenLLMetry team along with the CNCF OpenTelemetry working group has been doing some great work standardizing semantic naming conventions and we are starting to adopt the same set of standards.

[1] https://github.com/Scale3-Labs/langtrace [2] https://github.com/open-telemetry/semantic-conventions/blob/...

Langtrace core maintainer here. We capture input/output/total tokens for both streaming and non streaming across various LLM providers. For streaming we calculate using tiktoken library as OpenAI does not show that with their outputs. Cost is calculated on the client side as it may keep changing. But all you need is the cost table for the llm vendor and you can get the cost for token usage

Shameless plug. We are building exactly this. Fully open source and open telemetry standard tracing with a visualization client that’s optimized for LLM application observability. Check it out here

- https://github.com/Scale3-Labs/langtrace - https://langtrace.ai/

Hey HN,

Ola and Karthik here. We are working on Langtrace (https://github.com/Scale3-Labs/langtrace), an open source, open telemetry based SDK and monitoring/evaluations client for LLM based applications. The SDK generates OTEL standard spans and traces for popular LLMs like OpenAI, Anthropic and Cohere, popular frameworks like Langchain and LlamaIndex and vectorDBs like ChromaDB and Pinecone.

The LLM monitoring/evaluations space has seen a number of products off late, both open source and closed source ones. However, a couple of things we have observed are: lack of standard spans and traces that creates vendor lock-in, different tools are optimized for solving different pain points - Evaluations, Prompt management, Datasets, etc.

We believe that adopting OpenTelemetry (OTEL) standard tracing not only allows teams to use the SDK without having to switch their observability client, but will also enable developers to develop tooling for any custom needs, such as capturing datasets, prompts, evaluations etc.

A note on what we have built so far:

[1] We have a Python and a TypeScript SDK and we have broken down the support for the LLM layer into 3 groups, LLMs, Frameworks and VectorDBs. Our SDKs are open telemetry compatible, can be installed and used independently and we also provide an option to pass custom exporters to export the traces and spans to any observability tool of your choice.

[2] An observability client that is hyper optimized for solving the unique pain points and challenges that come with LLM based apps like Evaluations, prompt iteration, datasets etc. We are SOC2 compliant and the client can also be self hosted if you have strict data privacy and protection requirements.

[3] Both the SDK and the client are fully open source. We are leaning on the community to try it out and provide us with feedback. A note about OpenTelemetry semantic conventions for LLMs - we would like to converge on standard names for trace attributes that follow the OTEL rules and are looking for feedback from experts here - https://github.com/Scale3-Labs/langtrace/discussions/71

We recognize that this project is early and there is a lot of room for improvement. We would love to hear your thoughts and feedback. Thanks!

Links:

[1] https://langtrace.ai/ [2] https://github.com/Scale3-Labs/langtrace [3] https://docs.langtrace.ai/introduction [4] https://langtrace.ai/blog/why-you-need-opentelemetry-based-o...