News for hackers. Stuff that matters.
Re:AI Flame Graphs by simpledood on Wednesday October 30, 2024 @01:14PM (#41997696)
I've tried using flame graphs, but in my view nothing beats the simplicity and succinctness of gprof output for quickly analyzing program bottlenecks.

https://ftp.gnu.org/old-gnu/Manuals/gprof-2.9.1/html_chapter...

For each function you know how much CPU is spent in the function itself, as opposed to child calls. All in a simple text file without the need for constantly scrolling, panning, and enlarging to get the information you need.

Re:AI Flame Graphs by _heimdall on Wednesday October 30, 2024 @02:13PM (#41998434)
> Imagine halving the resource costs of AI and what that could mean for the planet and the industry -- based on extreme estimates such savings could reduce the total US power usage by over 10% by 2030

The way this is phrased threw me off. It sounded to me like the author was comparing the power use of a more efficient LLM industry to US usage without LLMs and expecting it to be 10% lower.

Looking into the source linked with the claim, it doesn't even hold up when compared against how much power LLMs use today. The linked article raises an estimate that LLM power use could increase 15-23 times between 2023 and 2027, and that by 2030 LLMs could account for 20-25% of our total energy use.

Working that match backwards, the benefit the author is hailing as a success is that we would only increase energy use by say 7.5-11.5 times by 2027 and that in 2030 LLMs would only be 10% of the total energy use. That's not a win in my book, and doesn't account for the Jevan's Paradox problem where we would almost certainly just use all that efficiency gain to further grow LLM use compared to the 2030 prediction without the efficiency gains.

Re:AI Flame Graphs by Veserv on Wednesday October 30, 2024 @02:19PM (#41998494)
I do not really understand the mentioned difficulties with instruction profiling.

Are they saying it is hard to sample the stacks across the boundary? Are they saying it is hard to do so coherently because the accelerator engine is actually asynchronous so you need to do some sort of cross-boundary correlation?

However, they then talk about file systems and /proc representations which have nothing to do with the actual sampling process; only posing problems for the display of human-readable information. Many naive profiling, tracing, and logging implementations conflate these actions to their detriment; are they being conflated here or is it just a generic statement of the scope of problems?

Re:AI Flame Graphs by treefarmer on Wednesday October 30, 2024 @02:34PM (#41998644)
Would love it if it was available and open source so people could use it in their own projects (or on their own hardware), instead of only being available on Intel's AI Cloud. But cool idea and execution nevertheless!
Re:AI Flame Graphs by flamingspear on Wednesday October 30, 2024 @07:37PM (#42001803)
Yeah, would love to built-in support for this in PyTorch or TF
Re:AI Flame Graphs by ryao on Wednesday October 30, 2024 @05:20PM (#42000485)
Wow. Nice.
Re:AI Flame Graphs by impish9208 on Wednesday October 30, 2024 @05:35PM (#42000660)
Re:AI Flame Graphs by dan-robertson on Wednesday October 30, 2024 @06:23PM (#42001155)
Being able to ‘connect’ call stacks between python, c++, and the gpu/accelerator seems useful.

I wonder if this pushes a bit much towards flamegraphs specifically. They were an innovation when they were first invented and the alternatives were things like perf report, but now I think they’re more one tool among many. In particular, I think many people who are serious about performance often reach for things like pprof for statistical profiles and various traceing and trace-visualisation tools for more fine-grained information (things like bpftrace, systemtap, or custom instrumentation on the recording side and perfetto or the many game-development oriented tools on the visualisation (and sometimes instrumentation) side).

I was particularly surprised by the statement about intel’s engineers not knowing what to do with the flamegraphs. I read it as them already having tools that are better suited to their particular needs, because I think the alternative has to be that they are incompetent or, at best, not thinking about performance at all.

Lots of performance measuring on Linux is done through the perf subsystem and Intel have made a lot of contributions to make it good. Similarly, Intel have added hardware features that are useful for measuring and improving performance – an area where their chips have features that, at least on chips I’ve used, easily beat AMD’s offerings. This kind of plumbing is important and useful, and I guess the flamegraphs demonstrate that the plumbing was done.

Re:AI Flame Graphs by stefan_ on Wednesday October 30, 2024 @07:08PM (#42001546)
It's a bit weird, very much a "software optimization" approach. But looking at the flame graph, you couldn't tell a model running in FP32 from one in INT8, taking 3x the time and energy.
Re:AI Flame Graphs by bornfreddy on Thursday October 31, 2024 @01:11PM (#42008917)
And? This is an information trivially obtainable in a different way (e.g. using a stopwatch), while flamegraphs visualise where that time was spent, helping us to determine the parts that need to be optimised.
Re:AI Flame Graphs by yanniszark on Wednesday October 30, 2024 @07:46PM (#42001875)
Trying to find out more about this EU stall thing Brendan talks about. Is it instruction sampling that gives you the reason for the stall? Sounds like a pretty advanced hw functionality.
Re:AI Flame Graphs by r3tr0 on Wednesday October 30, 2024 @08:47PM (#42002281)
i am actually working on a platform that makes this sort of stuff easy. We use BPF under the hood and let you remotely deploy them across a cluster and visualize them.

Check us out: https://yeet.cx

Our current package index is a bit thin:

https://yeet.cx/discover

We have a ton in the pipeline and are going to add more in the coming weeks and release an SDK.

Re:AI Flame Graphs by saagarjha on Thursday October 31, 2024 @03:58AM (#42004415)
I never really liked flamegraphs much but I am going to put that aside for a bit and try to be as objective as possible.

I don't find the usecase presented here compelling. Cutting out the "yo we will save you $x billion in compute" costs the tools presented here seem to be…stacktraces for your kernels. Stacktraces that go from your Python code through the driver shim to the kernel and finally onto the GPU. Neat. I don't actually know very much about what Intel has in this area so perhaps this is a step forward for them? If so, I will always applaud people figuring out how to piece together symbols and whatnot to make profiling work.

However, I am still not very impressed. Sure, there are some workloads where it is nice to know that 70% of your time is spent in some GEMM. But I think the real optimization doesn't look like that all. For most "real" workloads, you already know the basics of how your kernels look and execute. Nobody is burning a million dollars an hour on a training run without knowing what each and every one of the important kernels are. Some of them were probably written by hand. Some might be written in higher-level PyTorch/Triton/JAX/whatever. Still others might be built on some general library. But the people who do this are not stupid, and they aren't going to be caught unawares that a random kernel has suddenly popped up on their flamegraph. They should already know what is there. And most of these tools have debugging facilities to dump intermediate state in forms that tools understand. Often this is incomplete and buggy, I know. But it's there and people do use them.

What these people are optimizing are things that flamegraphs do not show. That's things like latency in kernel launches, or synchronization overhead with the host. It's global memory traffic and warp stalls. Sure, the tools to profile this are immature compared to what the hyperscalers have for CPUs. But they are still present and used heavily: I don't buy the argument that knowing that your python code calls a kernel through __cuda12_ioctl_whatever is actually helpful. This seems like a solution searching for a problem, or maybe a basic diagnostic tool at best.

Re:AI Flame Graphs by bornfreddy on Thursday October 31, 2024 @01:23PM (#42009045)
> What these people are optimizing are things that flamegraphs do not show. That's things like latency in kernel launches, or synchronization overhead with the...

What OP is showing is an example of what can be shown on flamegraphs. They are a generic visualisation tool so if you want to include latency or whatever (financial cost maybe?) you are free to do it.

As for the rest, Intel is here providing tools for developers who would like to optimize the sw stacks on their platform. Invaluable if you would like to efficiency support non-NVidia hardware.

Re:AI Flame Graphs by saagarjha on Friday November 01, 2024 @03:54AM (#42014856)
Flamegraphs categorically cannot represent timeseries data. That's not what they are designed to do and they don't have a way to display it.
Re:AI Flame Graphs by bornfreddy on Saturday November 02, 2024 @03:35PM (#42028667)
That is not true, they definitely can represent some timeseries data in specific ways. But that's not even connected to what I said - I specifically mentioned latency which can be included in profiling data. Or am I misunderstanding what you are trying to say?
Re:AI Flame Graphs by saagarjha on Sunday November 03, 2024 @04:14AM (#42031979)
How would you indicate how long a kernel takes to launch in a flamegraph?