So it's important that vendors don't feel let off the hook to provide sane packaging just because there's an option to use a kitchen-sink container image they rebuild every day.

is this a real problem? exactly which embedded platform has a device that ROCm supports?

that's not embedded dev. if you

1. use underpowered devices to perform sophisticated tasks

2. using code/tools that operate at extremely high levels of "abstraction"

don't be surprised when all the inherent complexity is tamed using just more layers of "abstraction". if that becomes a problem for your cost/power/space budget then reconsider choice 1 or choice 2.

yes and in those instances you do not reach for pytorch/tensorflow on top of ubuntu on top of x86 with a discrete gpu and 32gb of ram. instead you reach for C and micro or some arm soc that supports baremetal or at most rtos. that's embedded dev.

so i'll repeat myself: if you want to run extremely high-level code then don't be "surprised pikachu" when your underpowered platform, that you chose due to concrete, tight budgets doesn't work out.

However, containers or Ubuntu Linux don’t perform great in that environment. Ubuntu is for desktops, containers are for cloud data centers. An offline stand-alone device is different. BTW, end users don’t typically aware that thing is a computer at all.

Personally, I usually pick Alpine or Debian Linux for similar use cases, bare metal i.e. without any containers.

Or at least you have some case as heaven never come. Or come just we do not aware now like internet. Can you need to use ibm to rub sna to provide a token ring based network. In 1980 …

Imagine and let us or they competite …

Tell that to their (much larger, more profitable, and better-funded) server org. This is far from true.

On Desktop you have to worry about things like... UIs, sound, Wine, etc.

https://github.com/RadeonOpenCompute/ROCm-docker/blob/master...

They also have some for Fedora. Looks like for this you need to install their repo:

    curl -sL https://repo.radeon.com/rocm/rocm.gpg.key | apt-key add - \
    && printf "deb [arch=amd64] https://repo.radeon.com/rocm/apt/$ROCM_VERSION/ jammy main" | tee /etc/apt/sources.list.d/rocm.list \
    && printf "deb [arch=amd64] https://repo.radeon.com/amdgpu/$AMDGPU_VERSION/ubuntu jammy main" | tee /etc/apt/sources.list.d/amdgpu.list \

So they are doing the packaging. There might even be documentation elsewhere for that type of setup.

I think my issue is more just with the mindset that it's okay to have one narrow slice of supported versions of everything that are "known to work together" and those are what's in the container and anything outside of those and you're immediately pooched.

This is not hypothetical btw, I've run into real problems around it with libraries like gproto, where tensorflow's bazel build pulls in an exact version that's different from the default one in nixpkgs, and now you get symbol conflicts when something tries to link to the tensorflow c++ API while linking to another component already using the default gproto. I know these problems are solveable with symbol visibility control and whatever, but that stuff is far from universal and hard to get right, especially if the person setting up the build rules for the library doesn't themselves use it in that type of heterogeneous environment (like, everyone at Google just links the same global proto version from the monorepo so it doesn't matter).

You need to ensure that there is only one version of any library used globally throughout the code and that the set of versions is compatible with each other, and preferably you also want everything to be built against the same toolchain with the same flags.

That usually means onboarding third-party libraries into your own build system.

Also, just to name it, it’s ridiculous that a specific graphics card manages to restrict the version of gproto that you’re using. You don’t have this problem with nvidia drivers, since cuda stuff is much less fiddly. AMD needs to pull a finger out and fix the bugs in their stack that make it so fragile like this.

Wanting to hold Python+C ecosystem more accountable is fair I think, at least from my own experience around half a year ago, Anaconda doesn't work and you need a Dockerfile for any sort of reproducibility, which can have issues since GPU with docker isn't that easy. And this means developers from the vendors working with Anaconda, for example, on solving the issue rather than just hoping for contributors to do it. If AMD were to make easy, reproducible builds without root or VM a reality, that would be reason enough to try their hardware. If not, hopefully Nvidia does and then there really would be no way across the moat for me at least.

A docker container is not really any different from any other process; the main difference is that it runs in a chroot pretty much.

But that has nothing to do with semver.

Semver gives you information about when when you can replace one version with another version. It doesn't promise that you can mix multiple versions together.

And you are mixing multiple versions if you are building against version x.y and linking against version x.(y+z).

If linking against a different version of the code breaks like that, that sounds like someone did semver wrong. If that happens a lot to you, then oh, I'm sorry about that happening.

Or rather, I install no versions of libraries because NixOS will put them all in the store in different folders, and will compile the executable to use the correct path (or patch the elf when needed)

it has an issue with pip because it's allergic to just randomly executing things as part of package management, but pip in general is wtf

Well, except for cuda. Which is a massive pile of proprietary software that people are using in production anyway.

I hear you. I think docker has been a plague on the quality of software. It's allowed "works for me" to become the norm, except it's now pronounced "works on the official docker image". It seems to be especially true in the ML sphere where compiling things is so temperamental that there's a lot of binaries being distributed.

Docker was meant to be a deployment platform, not a distribution medium.

Heh, if only. When working with F100's I've seen many terrible, terrible things.

Sadly if e.g. 95% of their users can use the container, then it could make economical sense to do it that way.

X86 cannot do 64 bit let us do this and that so the market can use only our cpu. Repeat with me x86-64 is impossible.

Not sure Apple is in this otherwise the real great competition come.

Man oh man where did we go wrong that cuda is the more compatible option over OpenCL?

AMD should just get it's shit together. This is ridiculous. Not the name, but the fact that you can only do FP64 on a GPU. Everybody is moving to FP16 and AMD is stuck on doubles?

The API level I could target was at least two or three versions behind the latest they have to offer.

But this is one of the great strengths of CUDA: I can develop a kernel on my workstation, my boss can demo it on his laptop and we can deploy it on Jetsons or the multi-gpu cluster with minimal changes and i can be sure that everything runs everywhere.

I just don't understand the details enough to understand why things are problematic without CUDA :(

Some architectures provide fast F16->F32 and F32->F16 conversion instructions so you can DIY the memory bandwidth saving - that always seemed reasonable to me, but I don't know if the AMD hardware people are/will go down that path.

Keeping all those FPUs busy is another problem and not easy, but in cases where it can be done FP32 is clearly desirable.

There’s getting something to “work”, which is often enough of a challenge with ROCm. Then there’s getting it to work well (next challenge).

Then there’s getting it to work as well as Nvidia/CUDA.

With Whisper, as one example, you should be running it with ctranslate2[0]. Of all the platforms on their supported list you won’t find ROCm.

When you really start to look around you’ll find that ROCm is (at best) still very much in the “get it to work (sometimes)” stage. In most cases it’s still a long way away from getting it to work well, and even further away from making it actually competitive with Nvidia for serious use cases and applications.

People get excited about the progress ROCm has made getting basic things to work with PyTorch and this is good - progress is progress. But saving 20% on the hardware when the equivalent Nvidia product is often somewhere between 5-10x as performant (at a fraction of the development time) because of vastly superior software support you realize pretty quickly Nvidia is actually a bargain compared to AMD.

I’m desperately rooting for Nvidia to have some actual competition but after six years of ROCm and my own repeated failed attempts to have it make any sense overall I’m only more and more skeptical that real competition in the space will come from AMD.

[0] - https://github.com/OpenNMT/CTranslate2

https://www.tomshardware.com/news/whisper-audio-transcriptio... is a good example of Nvidia having no excuses being double the price when it comes to Whisper inference, with 7900XTX being directly comparable with 4080, albeit with higher power draw. To be fair it's not using ROCm but Direct3D 11, but for performance/price arguments sake that detail is not relevant.

EDIT: Also using CTranslate2 as an example is not great as it's actually a good showcase why ROCm is so far behind CUDA: It's all about adapting the tech and getting the popular libraries to support it. Things usually get implemented in CUDA first and then would need additional effort to add ROCm support that projects with low amount of (possibly hobbyist) maintainers might not have available. There's even an issue in CTranslate2 where they clearly state no-one is working to get ROCm supported in the library. ( https://github.com/OpenNMT/CTranslate2/issues/1072#issuecomm... )

It easily is. See the benchmarks[0] from faster-whisper which uses Ctranslate2. That's 5x faster than OpenAI reference code on a Tesla V100. Needless to say something like a 4080 easily multiplies that.

> https://www.tomshardware.com/news/whisper-audio-transcriptio... is a good example of Nvidia having no excuses being double the price when it comes to Whisper inference, with 7900XTX being directly comparable with 4080, albeit with higher power draw. To be fair it's not using ROCm but Direct3D 11, but for performance/price arguments sake that detail is not relevant.

With all due respect to the author of the article this is "my first entry into ML" territory. They talk about a 5-10 second delay, my project can do sub 1 second times[1] even with ancient GPUs thanks to Ctranslate2. I don't have an RTX 4080 but if you look at the performance stats for the closest thing (RTX 4090) the performance numbers are positively bonkers - completely untouchable for anything ROCm based. Same goes for the other projects I linked, lmdeploy does over 100 tokens/s in a single session with LLama2 13b on my RTX 4090 and almost 600 tokens/s across eight simultaneous sessions.

> EDIT: Also using CTranslate2 as an example is not great as it's actually a good showcase why ROCm is so far behind CUDA: It's all about adapting the tech and getting the popular libraries to support it. Things usually get implemented in CUDA first and then would need additional effort to add ROCm support that projects with low amount of (possibly hobbyist) maintainers might not have available. There's even an issue in CTranslate2 where they clearly state no-one is working to get ROCm supported in the library. ( https://github.com/OpenNMT/CTranslate2/issues/1072#issuecomm... )

I don't understand what you're saying here. It (along with the other projects I linked here[2]) are fantastic examples of just how far behind the ROCm ecosystem is. ROCm isn't even on the radar for most of them as your linked issue highlights.

Things always get implemented in CUDA first (ten years in this space and I've never seen ROCm first) and ROCm users either wait months (minimum) for sub-par performance or never get it at all.

[0] - https://github.com/guillaumekln/faster-whisper#benchmark

[1] - https://heywillow.io/components/willow-inference-server/#ben...

[2] - https://news.ycombinator.com/item?id=37793635#37798902

One arcane detail is that whereas for PyTorch I have to set the env var HSA_OVERRIDE_GFX_VERSION to 10.3.0, getting it to run with whisper.cpp and llama.cpp requires setting it to 10.1.0. Good luck and may it cost you less hair than it did me.

NVIDIA fp32 (H100) has 2x more TFLOPS than AMD's fp32 (MI250) and AI doesn't need fp64 precision.