Looks fine, but it does not look like there is a automatic full function entry/exit trace, just sampling. The real benefit is when you do not even need to insert manual instrumentation points, you just hit run and you get a full system trace.

How well does the visualizer handle multi-TB traces? Usually pretty uncommon, but a 10-100 GB is not that hard to produce when doing full tracing.

Of note is that tracy is aimed at games, where sampling is often too expensive and not fine-grained enough. Hence the manual instrumenting.

For the Bevy game engine, we automatically insert tracy spans for each ECS system. In practice, users can just compile with the tracy feature enabled, and get a rough but very usable overview of which part of their game is taking a long time on the CPU.

I was talking about automatic instrumentation of every single function call by default. No manual instrumentation needed because everything is already instrumented.

To be fair, you do still want some manual instrumentation to correlate higher level things, but full trace everywhere answers most questions. You also want to be able to manually suppress calls for small functions since that can be performance relevant or distorting, but the point is “default on, manual off” over “default off, manual on”.

This would be unbelievably inefficient, game engines will be running hundreds of millions of functions per second. And if the code runs 10x slower with the trace active, then it's no longer sensible.

We use sampling for the cases where this level of detail is needed as it has lower overhead.

What use case did you find this useful for?

No, it is quite reasonable with efficient implementation. 10-50% overhead or so depending on function size distribution (since it is small fixed overhead per call, smaller functions result in a greater fraction of overhead). 10x for just function entry/exit recording would be grotesquely inefficient. You can do inefficient time travel debugging recording for less than that.

You do need to allocate a ton of memory for the recording buffer to record sizable amounts of trace data. GB per core-second of trace or so (ring buffer so you get to see the last N seconds, not you need to run for less than N seconds) but that is fine during development on normal dev machines.

It is useful for everything. Why would you not want full traces for everything? It is amazing. We use it for everything internally where I work. Or rather, it is part of it. We actually prefer full time travel debugging during development and automated testing (again, overhead is low enough) but it is not available for everything. So sometimes we are stuck with just traces.

Every function would be pretty overkill, but you can automatically install instrumentation on functions on demand or with a pre-determined user selection. You can even instrument on a line basis if your instrumentation is cheap enough. I've experimented with this in a VS extension I'm developing and I could easily browse through a non-trivial game codebase without causing noticeable performance overhead [1]. In the demo, the instrumentation is auto-installed on all functions within the file you opened. Obviously, this is just one project I was testing on but it shows that this type of tracing is feasible.

[1] https://www.youtube.com/watch?v=3PnVG49SFmU

How would you implement this may I ask? C++ does not have reflection in the language so you at best can do that by hooking into the running application, but C++ also aggressively inlines functions on anything except -O0 which mean your function call might never be a function call. Running at -O0 is alsp just generally a bad idea since many many instances of UB will never get caught.

The only way I can see doing this at compile time is with a compiler extension but then you are entirely locked in to 1 compiler.

Maybe if you compile with debug symbols but then well, you are shipping debug symbols...

gcc has "-finstrument-functions". This calls your code on every function entry and exit. I've used this previously for tracing as described here and to move memory-protection windows around based on the running code.

If you want something passable, most compilers have function prologue/epilogue hooks that you can write in plain code. Realistically, development and test systems are only going to target like 1 or 2 compilers, so it is not very much work. Unless you are distributing a source library and you want to get full-trace telemetry from customer systems in-development, that is probably all you really need to do.

If you want to really zoom, you need to get the hooks inlined and probably just written in straight assembly. You then need to optimize your binary format and recording system. You then need to start optimizing your memory bandwidth usage when that becomes the bottleneck. Your overhead in the end is basically limited by memory bandwidth; you can only shovel so many tens of GB/s of logging into memory. Note that persistence has likely been infeasible for the last 2 or so orders of magnitude; RAM is likely the only storage consistently fast enough for the data rates you want to generate when doing this.

Makes sense and that's for three detailed answer. May put that one on the list of things to play with in the future.

Alas, not for Linux. I've been using the unloved and mostly abandoned (and mostly awful) google perf tools on Linux. :-(

Hmm? I haven't used Tracy yet but the demo trace they show at the URL linked on GitHub[1] sure looks like a trace from an application running on Linux. The documentation[2] also seems to reference what you need to run it on Linux, and the NixOS derivation[3] also suggests it runs on at least Linux and macOS, and I was able to run several of the binaries including the UI and capture binary. I still hesitate to doubt you on this because I haven't figured out how one is supposed to actually use it but it surely seems to support Linux. (I will definitely find a use for this, it looks amazing.)

[1]: https://tracy.nereid.pl/

[2]: https://github.com/wolfpld/tracy/releases/latest/download/tr...

[3]: https://github.com/NixOS/nixpkgs/blob/nixos-24.05/pkgs/devel...

Tracy works fine on Linux! Just used it today.

OK. Thanks for straightening me out. I saw the .msi, but no linux packages. I will definitely check it out!

That github repo contains the sources. You can also compile tracy by cloning the repo.

I’ve used Tracy on Linux about two months ago. Works fine.