News for hackers. Stuff that matters.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by lacker on Thursday December 14, 2023 @03:00PM (#38646485)
The funny thing to me is that so much of the "AI software ecosystem" is just PyTorch. You don't need to develop some new framework and make it popular. You don't need to support a zillion end libraries. Just literally support PyTorch.

If PyTorch worked fine on Intel GPUs, a lot of people would be happy to switch.

Re:Intel CEO: 'The entire industry is motivated to eliminate... by shihab on Thursday December 14, 2023 @03:23PM (#38646746)
But you can't support Pytorch without a proper foundation in place. They don't need to support zillion _end_ libraries, sure, but they do need to have at least a very good set of standard libraries, equivalent of Cublas, Curand etc.

And they don't. My work recently had me working with rocRAND (Rocm's answer to Curand). It was frankly pretty bad- the design, performance (50% slower in places that don't make any sense because generating random numbers is not exact that complicated), and documentation (God it was awful).

Now, that's a small slice of the larger pie. But imagine if this trend continues for other libraries.

Re:Intel CEO: 'The entire industry is motivated to eliminate... by singhrac on Thursday December 14, 2023 @04:11PM (#38647311)
Generating random numbers is a bit complicated! I wrote some of the samplers in Pytorch (probably replaced by now) and some of the underlying pseudo-random algorithms that work correctly in parallel are not exactly easy... running the same PRNG with the same seed on all your cores will produce the same result, which is probably NOT what you want from your API.

But, to be honest, it's not that hard either. I'm surprised their API is 2x slower, Philox is 10 years old now and I don't think there's a licensing fee?

Re:Intel CEO: 'The entire industry is motivated to eliminate... by eternityforest on Thursday December 14, 2023 @06:19PM (#38648811)
I wonder if the next generation chips are going to just have a dedicated hardware RNG per-core if that's an issue?
Re:Intel CEO: 'The entire industry is motivated to eliminate... by lelanthran on Friday December 15, 2023 @09:03AM (#38654247)
Why bother?

It's not the generation that matters so much, it's the gathering of entropy, which comes from peripherals and not possible to generate on-die.

If you don't need cryptographically secure randomness, you still want the entropy for generating the seeds per thread/die/chip.

Re:Intel CEO: 'The entire industry is motivated to eliminate... by eternityforest on Friday December 15, 2023 @02:00PM (#38657356)
It absolutely is possible to generate entropy on-die, assuming you actually want entropy and not just a unique value that gets XORed with the seed, so you can still have repeatable seeds.

Pretty much every chip has an RNG which can be as simple as just a single free running oscillator you sample

Re:Intel CEO: 'The entire industry is motivated to eliminate... by lelanthran on Friday December 15, 2023 @04:38PM (#38659144)
> Pretty much every chip has an RNG which can be as simple as just a single free running oscillator you sample

Every chip may have some sort of noise to sample, but they are nowhere near good sources of entropy.

Entropy is not a binary thing (you either have it or don't), it's a spectrum and entropy gathered on-die is poor entropy.

Look, I concede that my knowledge on this subject is a bit dated, but the last time I checked there were no good sources of entropy on-die for any chip in wide use. All cryptographically secure RNGs depend on a peripheral to grab noise from the environment to mix into the entropy pool.

A free-running oscillator is a very poor source of entropy.

Re:Intel CEO: 'The entire industry is motivated to eliminate... by eternityforest on Friday December 15, 2023 @05:13PM (#38659470)
For non-cryptographic applications, a PRNG like xorshift reseeded by a few bits from an oscillator might be enough.

As I understand it, the reason they don't use on-chip RNGs by themselves isn't due to lack of entropy, it's because people don't trust them not to put a backdoor on the chips or to have some kind of bug.

Intel has https://en.m.wikipedia.org/wiki/RDRAND but almost all chips seem to now have some kind of RNG.

Re:Intel CEO: 'The entire industry is motivated to eliminate... by shihab on Thursday December 14, 2023 @07:20PM (#38649428)
> Generating random numbers is a bit complicated!

I know! I just wrote a whole paper and published a library on this!

But really, perhaps not as much as many from outside might think. The core of a Philox implementation can be around 50 lines of C++ [1], with all the bells and whistles maybe around 300-400. That implementation's performance equals CuRAND's , sometimes even surpasses it! (the API is designed to avoid maintaining any rng states on device memory, something curand forces you to do).

> running the same PRNG with the same seed on all your cores will produce the same result

You're right. Solution here is to utilize multiple generator objects, one per thread, ensuring each produces statistically independent random streams. Some good algorithms (Philox for example), allow you to use any set of unique values as seeds for your threads (e.g. thread id).

[1] https://github.com/msu-sparta/OpenRAND/blob/main/include/ope...

Re:Intel CEO: 'The entire industry is motivated to eliminate... by carterschonwald on Thursday December 14, 2023 @08:34PM (#38650021)
Cool! I’ll have a lookseee. I’ve my own experiments in this space.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by paulmd on Thursday December 14, 2023 @11:28PM (#38650995)
for GPGPU, the better approach is CBRNG like random123.

https://github.com/DEShawResearch/random123

if you accept the principles of encryption, then the bits of the output of crypt(key, message) should be totally uncorrelated to the output of crypt(key, message+1). and this requires no state other than knowing the key and the position in the sequence.

moreover, you can then define the key in relation to your actual data. the mental shift from what you're talking about is that in this model, a PRNG isn't something that belongs to the executing thread. every element can get its own PRNG and keystream. And if you use a contextually-meaningful value for the element key, then you already "know" the key from your existing data. And this significantly improves determinism of the simulation etc because PRNG output is tied to the simulation state, not which thread it happens to be scheduled on.

(note that the property of cryptographic non-correlation is NOT guaranteed across keystreams - (key, counter) is NOT guaranteed to be uncorrelated to (key+1, counter), because that's not how encryption usually is used. with a decent crypto, it should still be very good, but, it's not guaranteed to be attack-resistant/etc. so notionally if you use a different key index for every element, element N isn't guaranteed to be uncorrelated to element N+1 at the same place in the keystream. If this is really important then maybe you want to pass your array indexes through a key-spreading function etc.)

there are several benefits to doing it like this. first off obviously you get a keystream for each element of interest. but also there is no real state per-thread either - the key can be determined by looking at the element, but generating a new value doesn't change the key/keystream. so there is nothing to store and update, and you can have arbitrary numbers of generators used at any given time. Also, since this computation is purely mathematical/"pure function", it doesn't really consume any memory-bandwidth to speak of, and since computation time is usually not the limiting element in GPGPU simulations this effectively makes RNG usage "free". my experience is that this increases performance vs CuRand, even while using less VRAM, even just directly porting the "1 thread = 1 generator" idiom.

Also, by storing "epoch numbers" (each iteration of the sim, etc), or calculating this based on predictions of PRNG consumption ("each iteration uses at most 16 random numbers"), you can fast-forward or rewind the PRNG to arbitrary times, and you can use this to lookahead or lookback on previous events from the keystream, meaning it serves as a massively potent form of compression as well. Why store data in memory and use up your precious VRAM, when you could simply recompute it on-demand from the original part of the original keystream used to generate it in the first place? (assuming proper "object ownership" ofc!) And this actually is pretty much free in performance terms, since it's a "pure function" based on the function parameters, and the GPGPU almost certainly has an excess of computation available.

In the extreme case, you should be able to theoretically "walk" huge parts of the keystream and find specific events you need, even if there is no other reference to what happened at that particular time in the past. Like why not just walk through parts of the keystream until you find the event that matches your target criteria? Remember since this is basically pure math, it's generated on-demand by mathing it out, it's pretty much free, and computation is cheap compared to cache/memory or notarizing.

(ie this is a weird form of "inverted-index searching", analogous to Elastic/Solr's transformers and how this allows a large number of individual transformers (which do their own searching/indexing for each query, which will be generally unindexable operations like fulltext etc) to listen to a single IO stream as blocks are broadcast from the disk in big sequential streaming batches. Instead of SSD batch reads you'd be aiming for computation batch reads from a long range within a keystream.)

Anyway I don't know how much that maps to your particular use-case but that's the best advice I can give. Procedural generation using a rewindable, element-specific keystream is a very potent form of compression, and very cheap. But, even if all you are doing is just avoiding having to store a bunch of CuRand instances in VRAM... that's still an enormous win even if you directly port your existing application to simply use the globalThreadId like it was a CuRand stateful instance being loaded/saved back to VRAM. Like I said, my experience is that because you're changing mutation to computation, this runs faster and also uses less VRAM, it is both smaller and better and probably also statistically better randomness (especially if you choose the "hard" algorithms instead of the "optimized" versions like threefish instead of threefry etc).

That is the reason why you shouldn't do the "just download random numbers", as a sibling comment mentions (probably a joke) - that consumes VRAM, or at least system memory (and pcie bandwidth). and you know what's usually way more available as a resource in most GPGPU applications than VRAM or PCIe bandwidth? pure ALU/FPU computation time.

buddy, everyone has random numbers, they come with the fucking xbox. ;)

Re:Intel CEO: 'The entire industry is motivated to eliminate... by paulmd on Friday December 15, 2023 @12:24AM (#38651277)
thinking this through a little bit, you are launching a series of gradient-descent work tasks, right? taskId is your counter value, weightIdx is your key value (RNG stream). That's how I'd port that. Ideally you want to define some maximum PRNG usage for each stage of the program, which allows you to establish fixed offsets from the epoch value for a given event. Divide your keystream in whatever advantageous way, based on (highly-compressible) epoch counters and event offsets from that value.

in practice, assuming a gradient-descent event needs a lot of random numbers, having one keystream for a single GD event might be too much and that's where key-spreading comes in. if you take the "weightIdx W at GradientDescentIdx G" as the key, you can have a whole global keystream-space for that descent stage. And the key-spreading-function lets you go between your composite key and a practical one.

https://en.wikipedia.org/wiki/Key_derivation_function

(again, like threefry, there is notionally no need for this to be cryptographically secure in most cases, as long as it spreads in ways that your CBRNG crypto algorithm can tolerate without bit-correlation. there is no need to do 2 million rounds here either etc. You should actually pick reasonable parameters here for fast performance, but good enough keyspreading for your needs.)

I've been out of this for a long time, I've been told I'm out of date before and GPGPUs might not behave exactly this way anymore, so please just take it in the spirit it's offered, can't guarantee this is right but I've specifically gazed into the abyss the CuRand situation a decade ago and this was what I managed to come up with. I do feel your pain on the stateful RNG situation, managing state per-execution-thread is awful and destroys simulation reproducibility, and managing a PRNG context for each possible element is often infeasible. What a waste of VRAM and bandwidth and mutation/cache etc.

And I think that cryptographic/pseudo-cryptographic PRNG models are frankly just a much better horse to hook your wagon to than scientific/academic ones, even apart from all the other advantages. Like there's just not any way mersenne twister or w/e is better than threefish, sorry academia

--

edit: Real-world sim programs are usually very low-intensity and have effectively unlimited amounts of compute to spare, they just ride on bandwidth (sort/search or sort/prefix-scan/search algorithms with global scope building blocks often work well).

And tbh that's why tensor is so amazing, it's super effective at math intensity and computational focus, and that's what GPUs do well, augmented by things like sparse models etc. Make your random not-math task into dense or sparse (but optimized) GPGPU math, plus you get a solution (reasonable optimum) to an intractible problem in realtime. The experienced salesman usually finds a reasonable optimum, but we pay him in GEMM/BLAS/Tensor compute time instead of dollars.

Sort/search or sort/prefix-sum/search often works really well in deterministic programs too. Do you ever have a "myGroup[groupIdx].addObj(objIdx) stage? that's a sort and prefix-sum operation right there, and both of those ops run super well on GPGPU.

Re:Intel CEO: 'The entire industry is motivated to eliminate... by lumost on Thursday December 14, 2023 @03:33PM (#38646864)
Folks also underestimate how complex these libraries are. There are dozens of projects to make BLAS alternatives which give up after ~3-6 months when they realize that this project will take years to be successful.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by eternityforest on Thursday December 14, 2023 @06:21PM (#38648832)
How does that work? Why not pick up where the previous team left off instead of everyone starting new ones? Or are they all targeting different backends and hardware?
Re:Intel CEO: 'The entire industry is motivated to eliminate... by haltist on Thursday December 14, 2023 @06:43PM (#38649052)
It's a compiler problem and there is no money in compilers [1]. If someone made an intermediate representation for AI graphs and then wrote a compiler from that intermediate format into whatever backend was the deployment target then they might be able to charge money for support and bug fixes but that would be it. It's not the kind of business anyone wants to be in so there is no good intermediate format and compiler that is platform agnostic.

1: https://tinygrad.org/

Re:Intel CEO: 'The entire industry is motivated to eliminate... by whatshisface on Thursday December 14, 2023 @11:09PM (#38650895)
JAX is a compiler.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by haltist on Friday December 15, 2023 @12:07AM (#38651193)
So are TensorFlow and PyTorch. All AI/ML frameworks have to translate high-level tensor programs into executable artifacts for the given hardware.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by nradov on Thursday December 14, 2023 @06:18PM (#38648794)
Instead of generating pseudorandom numbers you can just download files of them.

https://archive.random.org/

Re:Intel CEO: 'The entire industry is motivated to eliminate... by hcrean on Thursday December 14, 2023 @09:16PM (#38650372)
Or you could just re-use the same number; no one can prove it is not random.

https://xkcd.com/221/

Re:Intel CEO: 'The entire industry is motivated to eliminate... by slavik81 on Thursday December 14, 2023 @07:57PM (#38649714)
If you haven't already, please consider filing issues on the rocrand GitHub repo for the problems you encountered. The rocrand library is being actively developed and your feedback would be valuable for guiding improvements.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by shihab on Friday December 15, 2023 @08:00PM (#38660770)
Appreciate it, will do.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by jart on Friday December 15, 2023 @02:49AM (#38651966)
I honestly don't see why it's so hard. On my project we wrote our own gemm kernels from scratch so llama.cpp didn't need to depend on cublas anymore. Only took a few days and a few hundred lines of code. We had to trade away 5% performance.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by soulbadguy on Friday December 15, 2023 @04:00AM (#38652362)
For a given set kernels, and a limited set of architectures, the problem is relatively easy.

But covering all the important kernels acros all the crazy architecture out there and with relatively good performance and numerical accuracy ... Much harder

Re:Intel CEO: 'The entire industry is motivated to eliminate... by rightbyte on Friday December 15, 2023 @04:13AM (#38652417)
> generating random numbers

You can't bench implementations of random numbers against each other purely on execution speed.

A better algorithm (better statistical properties) will be slower.

Re:Intel CEO: 'The entire industry is motivated to eliminate... by codetrotter on Friday December 15, 2023 @05:32AM (#38652813)
I have the fastest random number generator in the world. And it works in parallel too!

https://i.stack.imgur.com/gFZCK.jpg

Re:Intel CEO: 'The entire industry is motivated to eliminate... by shihab on Friday December 15, 2023 @02:51PM (#38658015)
Yeah. In this instance, I was talking about the same algorithm (phillox), the difference is purely in implementation.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by singhrac on Thursday December 14, 2023 @04:13PM (#38647334)
Just to point out it does, kind of: https://github.com/intel/intel-extension-for-pytorch

I've asked before if they'll merge it back into PyTorch main and include it in the CI, not sure if they've done that yet.

In this case I think the biggest bottleneck is just that they don't have a fast enough card that can compete with having a 3090 or an A100. And Gaudi is stuck on a different software platform which doesn't seem as flexible as an A100.

Re:Intel CEO: 'The entire industry is motivated to eliminate... by spacemanspiff01 on Thursday December 14, 2023 @04:33PM (#38647551)
They could compete on ram, if the software was there. Just having a low cost alternative to the 4060ti would allow them to break into the student/hobbies/open source market.

I tried the a770, but returned it. Half the stuff does not work. They have the CPU side and GPU development on different branches (GPU seems to be ~6 months behind CPU) and often you have to compile it yourself, (if you want torchvision or torchaudio) it also currently on 2.0.1 of pytorch so somewhat lagging, and does not have most of the performance analysis software available. You also, do need to modify your pytorch code, often more than just replacing cuda for xpu as the device. They are also doing all development internally, then pushing intermittently to public. A lot of this would not be as bad if there was a better idea of feature timeline, or if they made their CI public. (Trying to build it myself involved a extremely hacky bash script, that inevitably failed halfway through.)

Re:Intel CEO: 'The entire industry is motivated to eliminate... by selfhoster11 on Thursday December 14, 2023 @06:20PM (#38648817)
The amount of VRAM is the absolute killer USP for the current large AI model hobbyist segment. Something that had just as much VRAM as a 3090 but at half the speed and half the price would sell like hot cakes.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by eyegor on Thursday December 14, 2023 @08:21PM (#38649920)
You are describing the ebay market for used nvidia tesla cards. The k80, p40, or m40 are widely available and sell for ~$100 with 24gb vram. The m10 even has 32gb! The problem for ai hobbyists is it won't take long to realize how many apis use the "optical flow" pathways and so on nvidia they'll only run at acceptable speeds on rtx hardware, assuming they run at all. Cuda versions are pinned to hardware to some extent.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by icelancer on Thursday December 14, 2023 @09:29PM (#38650462)
Yep. I have a fleet of P40s that are good at what they do (Whisper ASR primarily) but anything even remotely new... nah. fp16 support is missing so you need P100 cards, and usually that means you are accepting 16GB of VRAM rather than 24GB.

Still some cool hardware.

Re:Intel CEO: 'The entire industry is motivated to eliminate... by muxr on Friday December 15, 2023 @08:14AM (#38653836)
For us hobbyists used 3090 or new 7900xtx seem to be the way. But even then you still need to build a machine with 3 or 4 of these GPUs to get enough VRAM to play with big models.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by icelancer on Friday December 15, 2023 @02:36PM (#38657820)
For sure - our prod machine has 6x RTX 3090s on some old cirrascale hardware. But P40s are still good for last-gen models. Just nothing new unfortunately.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by selfhoster11 on Friday December 15, 2023 @03:15AM (#38652112)
Out of these three, only P40 is worth the effort to get running vs the capabilities they offer. That's also before considering that other than software hacks or configuration tweaks, those cards require specialised cooling shrouds for adequate cooling in tower-style cases.

If your time or personal energy is worth >$0, these cards work out to much more than $100. And you can't even file the time burnt on getting them to run as any kind of transferable experience.

That's not to say I don't recommend getting them - I have a P4 family card and will get at least one more, but I'm not kidding myself that the use isn't very limited.

Re:Intel CEO: 'The entire industry is motivated to eliminate... by icelancer on Friday December 15, 2023 @02:39PM (#38657864)
The K80 is literally worth < $0 given how ridiculous it is to configure and use.

P40s are definitely the best value along with P100s. I still think there is a good amount of value to be extracted from both of those cards, especially if you are interested in using Whisper ASR, video transcoding, or CUDA models that were relevant before LLMs (a time many people have forgotten apparently).

Re:Intel CEO: 'The entire industry is motivated to eliminate... by singhrac on Thursday December 14, 2023 @06:51PM (#38649153)
This is pretty disappointing to hear. I’m really surprised they can’t even get a clean build script for users, let alone integrate into the regular Pytorch releases.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by paulmd on Friday December 15, 2023 @12:37AM (#38651338)
oh no, I just bought a refurbed A770 16GB for tinkering with GPGPU lol. It was $220, return?
Re:Intel CEO: 'The entire industry is motivated to eliminate... by zozbot234 on Thursday December 14, 2023 @03:48PM (#38647024)
PyTorch includes some Vulkan compat already (though mostly tested on Android, not on desktop/server platforms), and they're sort of planning to work on OpenCL 3.0 compat, which would in turn lead to broad-based hardware support via Mesa's RustiCL driver.

(They don't advertise this as "support" because they have higher standards for what that term means. PyTorch includes a zillion different "operators" and some of them might be unimplemented still. Besides performance is still lacking compared to CUDA, Rocm or Metal on leading hardware - so only useful for toy models.)

Re:Intel CEO: 'The entire industry is motivated to eliminate... by latchkey on Thursday December 14, 2023 @03:13PM (#38646630)
This is a big reason why AMD did this deal with PyTorch...

https://pytorch.org/blog/experience-power-pytorch-2.0/

Re:Intel CEO: 'The entire industry is motivated to eliminate... by ndneighbor on Thursday December 14, 2023 @06:56PM (#38649203)
OneAPI isn't bad for PyTorch, the performance isn't there yet but you can tell it's an extremely top priority for Intel.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by flakiness on Thursday December 14, 2023 @07:18PM (#38649406)
Intel has to do it by themselves. NVIDIA just lets Meta/OpenAI/Google engineers do it for them. Such a handicapped fight.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by joe_the_user on Thursday December 14, 2023 @09:18PM (#38650391)
That's because CUDA is a clear, well-functioning library and Intel has no equivalent. It makes any "you just have to get Pytorch working" a little less plausible.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by seanhunter on Friday December 15, 2023 @12:16AM (#38651240)
It wasn’t always like this. Nvidia did the initial heavy lifting to get cuda off the ground to a point where other people could use it.
Re:Intel CEO: 'The entire industry is motivated to eliminate... by seanhunter on Friday December 15, 2023 @12:15AM (#38651231)
But this is the the thing. Speaking as someone who dabbles in this area rather than any kind of expert, it’s baffling to me that people like Intel are making press releases and public statements rather than (I don’t know) putting in the frikkin work to make performance of the one library that people actually use decent.

You have a massive organization full of gazillions of engineers many of whom are really excellent. Before you open your mouth in public and say something is a priority, deploy a lot of them against this and manifest that priority by actually doing the thing that is necessary so people can use your stuff.

It’s really hard to take them seriously when they haven’t (yet) done that.

Re:Intel CEO: 'The entire industry is motivated to eliminate... by pas on Friday December 15, 2023 @05:24AM (#38652765)
You know how it works. The same busybodies who are putting out this useless noise releases are the ones who squandered Intel's lead, and now are patting themselves on the back for figuring out that with this they'll again be on top for sure!

There was a post on HN a few months ago about how Nvidia's CEO still has meetings with engineers in the trenches. Contrast that with what we know of Intel, which is not much good, and a lot of bad. (That they are notoriously not-well-paying, because they were riding on their name recognition.)