edenfed

Speaking for Odigos (disclosure: I’m the creator), here are two significant differences between us and the other mentioned players:

- Accurate distributed traces with eBPF, including context propagation. Without going into other tools, I highly recommend trying to generate distributed traces using any other eBPF solution and observing the results firsthand.

- We are agent-only. Our data is produced in OpenTelemetry format, allowing you to integrate it seamlessly with your existing observability system.

I hope this clarifies the differences.

Definitely can relate, this is why I started an open-source project that focus on making OpenTelemetry adoption as easy as running a single command line: https://github.com/odigos-io/odigos

You can absolutely use just the OTel APIs and use something else besides the OTel SDK. Here is a blog post about how we did it with eBPF: https://odigos.io/blog/Integrating-manual-and-auto

Interesting idea. I think that as long as you able to do processing, serializing and delivery in other process and save this work from your application runtime you should see great performance

By dropped data do you mean by exceeding the size of the allocated ring buffer/perf buffer? If so this is configurable by the user, so you can adjust is according to the expected load

You can enrich the spans created by eBPF by using OpenTelemetry APIs as usual, the eBPF instrumentation is a replacement for the instrumentation SDK. The eBPF program will detect the data recorded via the APIs and will add it to the final trace combining both automatic and manually created data.

I don’t have a lot of experience using dtrace, but AFAIK the big advantage of eBPF over dtrace is that you do not need to instrument your application with static probes during coding.

Thank you for reporting will fix ASAP

We already solved compiled languages (Go, C, Rust) and JIT languages (Java, C#). Interpreted languages (Python, JS) are the only ones left, hopefully we will solve these as well soon. The big challenge is supporting all the different runtimes, once that is solved implementing support for different protocols / open-source libraries is not as complicated.

We have a solution for virtual thread as well. Currently working on a blog post describing exactly how. Will update once releases

eBPF instrumentation does not require code changes, redeployment or restart to running applications.

We are constantly adding more language support for eBPF instrumentation and are aiming to cover the most popular programming languages soon.

Btw, not sure that sampling is really the solution to combat overhead, after all you probably do want that data. Trying to fix production issue when the data you need is missing due to sampling is not fun

We are currently supporting just Kubernetes environments. docker-compose, VMs, and Serverless are on our roadmap and will be ready soon

We also thinking on implementing fallback mechanism to automatically propagate context on the same goroutine if context.Context is not passed

Thanks for the valuable feedback! We used a constant throughout of 10,000 rps. The exact testing setup can be found under “how we tested”.

I think the example you gave for the lock used by Prometheus library is a great example why generation of traces/metrics is a great fit for offloading to different process (an agent).

Patchyderm looks very interesting however I am not sure how you can generate distributed traces based on metrics, how do you fill in the missing context propagation?

Our way to deal with eBPF root requirements is to be transparent as possible. This is why we donated the code to the CNCF and developing as part of the OpenTelemetry community. We hope that being open will make users trust us. You can see the relevant code here: https://github.com/open-telemetry/opentelemetry-go-instrumen...

I recommend watching Gil Tene’s talk, I think he explains the math better than I do: https://www.youtube.com/watch?v=lJ8ydIuPFeU

Logs are easy and familiar API for adding additional data to your traces. They still have their place, Odigos is just adding much more context.

Nothing special, if you are working on Kubernetes its as easy as running `odigos install` CLI and pointing to your current monitoring system.

I think eBPF has also great potential to help JVM-based languages. Especially around performance aspects even comparing to the current java agents which use bytecode manipulation.

It depends on the programming language being instrumented. For Go we are assuming the context.Context object is passed around between different functions or goroutines. For Java, we are using a combination of ThreadLocal tracing and Runnable tracing to support use cases like reactive and multithreaded applications.

Can you try again please? It works well for me

Disclaimer: I am one of the maintainers

Many comments complain about the complexity of using OpenTelemetry, I recommend checking out Odigos, an open-source project which makes working with OpenTelemetry much easier: https://github.com/keyval-dev/odigos

We combine OpenTelemetry and eBPF to instantly generate distributed traces without any code changes.

Hi SergeAx, I am building Odigos, which do exactly what you asked :) We combine OpenTelemetry and eBPF to automatically generate and deliver distributed traces, metrics and logs to Grafana, DataDog and other 15+ destinations. Check it out here: https://github.com/keyval-dev/odigos