BigRedEye

I think this single fact is a major source of enshittification in large software products, especially in the era of ML/AI. If your quality is 99%, it sounds like "you have solved your task", but in reality there is a long tail that over time affects nearly every customer.

I've seen this so many times. 99% of search results are good (so within 100 queries you'll hit at least one bad result with p≈0.63), 99% of dashboard panes load normally (so a dashboard with 20 panes is broken in nearly 1 in 5 loads), and so on. If your LLM gets 99% of tool calls right, nearly every session will contain a malformed tool call.

Probabilities are hard for humans, probably.

At Perforator, we also started from Google's beautiful pprof, but then eliminated all nested repeated fields, converging to https://github.com/yandex/perforator/blob/main/perforator/pr.... Repeated fields in protobufs are really memory & CPU hungry.

This layout allows us to quickly merge hundreds of millions of samples into a single profile. The only practical limit is protobuf's 2GB message size cap.

I believe this is a case of convergent invention – the idea of pushing DWARF/.eh_frame unwinding into eBPF seems to have occurred to several people around the same time. For example, there's a working implementation discussed as early as March 2021: https://github.com/iovisor/bcc/issues/1234#issuecomment-7875...

At Yandex we have a similar profiler that supports native languages seamlessly, with addition to Python/Java: https://github.com/yandex/perforator. It's exciting to see new profilers from big players!

Great question! Perforator indeed looks similar to Pyroscope. However, we think that the closest existing solutions are https://parca.dev, closed-source Google Wide Profiling, and, speaking of the agent, the beautiful OpenTelemetry eBPF profiler. The main technical differences with Pyroscope we see are:

- Pyroscope's Java support is superior as of now because Pyroscope offloads it to the amazing async-profiler.

- Pyroscope expects native binaries to be compiled with frame pointers: https://grafana.com/docs/pyroscope/latest/configure-client/g.... This is often not the case, and that's the problem we've tried to solve with Perforator. Perforator uses .eh_frame, which is nearly universal and does not impose additional requirements on compiled binaries.

- Pyroscope symbolizes using symtab: https://grafana.com/docs/pyroscope/latest/configure-client/g.... We use DWARF/GSYM to get as correct and verbose stacks as possible (we benchmark our stacks against stacks from gdb).

- Pyroscope symbolizes profiles on an agent, while Perforator symbolizes profiles offline, greatly reducing symbolization costs and agent's overhead. It seems Pyroscope is heading toward the same architecture we use: https://github.com/grafana/pyroscope/pull/3799.

- Perforator can be (and should be!) run as a standalone replacement for perf record.

- Perforator supports sPGO profiles.

In summary, we try to implement native profiling almost perfectly. It's worth noting that Pyroscope is a mature, well-established product that integrates excellently with the Grafana ecosystem. We have just focused on different things: our focus has been on optimizing native code profiling and making it as accurate and low-overhead as possible.

It depends on the event that was sampled to generate the profiles. For example, if you sample instructions by collecting a stack trace every N instructions, you won't actually see foo() burning the CPU. However, if you look at CPU cycles, foo() will be very noticeable. Internally, we use sPGO profiles from sampling CPU cycles, not instructions.

A short discussion can be found here: https://news.ycombinator.com/item?id=42888185

Yes, existing limitations apply. Without hardware LBR support, we cannot provide sPGO profiles. However, the basic profiling should work fine.

Yes. Although we are studying CSSPO, which uses a mixed (LBR + software-sampled stacks) approach.

Hey HN! We are happy to share Perforator – our internal cluster-wide profiler with great support for native languages and a built-in AutoFDO pipeline to simplify sPGO builds. Perforator allows you to profile most binaries without having to recompile or adjust the build process. We use it at Yandex to profile each pod inside a large cluster at modest speed (99Hz), collecting petabytes of profiles every day.

Inspired by Google-Wide Profiling, we started continuous profiling years ago with simple tools like poormansprofiler.org. With the rise of eBPF, we came up with a simple and elegant solution providing detailed profiles without noticeable overhead. Pretty wild when you can see the guts of your production binaries in a flamegraph without them even noticing.

Some technical details:

- Our main contribution is infrastructure for continuous PGO using AutoFDO. Google and Meta have done tremendous work on building PGO infrastructure, and we made the last missing piece of the puzzle to make this work well and scalable.

- Native binaries are profiled through eh_frame analysis, interpreted/JIT-compiled languages are profiled through perf-pid.map or hardcoded structure offsets.

- We render profiles in multiple ways, the most common one is a fast implementation of FlameGraphs, rendering 1M frames in 100ms.

- We provide Helm charts to easily deploy Perforator on your k8s cluster.

- You can use Perforator in standalone mode as a replacement for perf record.

I'd love to answer your questions about the tool!

22 minutes for a medium-sized repo is probably slow enough to optimize.