News for hackers. Stuff that matters.
Re:Triton: Open-Source GPU Programming for Neural Networks by shubuZ on Wednesday July 28, 2021 @01:09PM (#27986188)
I have found writing CUDA code is much simpler than writing correct multi-threaded AVX2/AVX-512 code.
Re:Triton: Open-Source GPU Programming for Neural Networks by dragontamer on Wednesday July 28, 2021 @01:48PM (#27986875)
If you need CPU-side SIMD, then try ispc: https://ispc.github.io/

Its pretty much the OpenCL-model, except it compiles into AVX2 code / AVX512 code. Very similar to CUDA / OpenCL style programming. Its not single-source like CUDA, but it largely accomplishes the programming model IMO.

Re:Triton: Open-Source GPU Programming for Neural Networks by gnufx on Wednesday July 28, 2021 @02:56PM (#27988090)
Why not a standard? OpenMP is more than pretty much C(++) and Fortran, and has offload inspired by the needs of the Sierra supercomputer.
Re:Triton: Open-Source GPU Programming for Neural Networks by dragontamer on Wednesday July 28, 2021 @03:03PM (#27988193)
OpenMP 4.5+ looks very promising to me (particularly the "simd" keyword associated with for-loops). But open-source implementations of OpenMP are somewhat lackluster... at least last time I checked it out.

Maybe its time I revisit it. I've always thought the OpenMP spec was being written by highly competent programmers. They seem to "get" what is needed. But the question is if I can get my hands on any OpenMP implementation to actually play with.

Not all of us can afford IBM's compiler suite!

Re:Triton: Open-Source GPU Programming for Neural Networks by gnufx on Wednesday July 28, 2021 @05:14PM (#27989803)
I should really find time to do comparisons with the compilers to hand, including XL, on the NAS benchmarks, as I've never seen that, though it must have been done. I think we have a "community" version of XL, i.e. no support, like basically everything else. I wasn't aware there was anything much wrong with GCC and libgomp or libomp, but then I haven't measured.
Re:Triton: Open-Source GPU Programming for Neural Networks by jpf0 on Wednesday July 28, 2021 @03:08PM (#27988253)
LLVM has an openMP implementation
Re:Triton: Open-Source GPU Programming for Neural Networks by dragontamer on Wednesday July 28, 2021 @03:10PM (#27988281)
The task-based parallelism in LLVM leaves much to be desired however. Ideally, you'd want a more efficient implementation.

But yeah, good enough to play with. But maybe not good enough to achieve high levels of performance. The SIMD stuff is probably simple enough to implement... maybe I should checkout how well LLVM works with OMP SIMD keywords.

Re:Triton: Open-Source GPU Programming for Neural Networks by jpf0 on Wednesday July 28, 2021 @05:00PM (#27989641)
Can you comment on experience (or contact me) regarding implementation efficiency? We have recently implemented task-based parallelism in the J language with openMP[0]. Improvements or critiques are appreciated. SIMD instructions there have been coded directly rather than via pragmas.

[0] https://www.monument.ai/m/parallel

Re:Triton: Open-Source GPU Programming for Neural Networks by dragontamer on Wednesday July 28, 2021 @05:26PM (#27989956)
I can't say that my critiques are based off of personal experience. But mostly about microbenchmarks I've read that other people have talked about. I am probably a bit out of date, since its been a while since I last played with OpenMP.

I'm looking at the benchmarks I used to look at, and they're all from 2014 or earlier. So maybe I really should double-check modern implementations. We all know GCC 4.x and LLVM 3.x are an eternity ago, so I probably should revisit their performance.

For example: https://www.phoronix.com/scan.php?page=article&item=llvm_cla...

And back then, it was pretty well known that OpenMP implementations were slower than commercial (such as Intel ICC or IBM's OpenMP implementation).

Re:Triton: Open-Source GPU Programming for Neural Networks by gnufx on Wednesday July 28, 2021 @05:18PM (#27989852)
In LLVM or in libomp? I don't know what omp simd is likely to get you over autovectorization. I know of cases where it was thought necessary (-fopenmp-simd, without -fopenmp) but wasn't with recent GCC.
Re:Triton: Open-Source GPU Programming for Neural Networks by dragontamer on Wednesday July 28, 2021 @05:41PM (#27990100)
Autovectorization has issues with function calls.

"#pragma omp declare simd" applies over a function call, which then allows that function to be used inside of a "#pragma omp for simd" loop.

A few keywords here and there really help the autovectorizer achieve closer to CUDA-like environments (like... actually having your SIMD code extend "through" a function call, so you can start splitting up the work a bit better).

EDIT: Here's an example from Intel's ICC: https://software.intel.com/content/www/us/en/develop/documen...

Re:Triton: Open-Source GPU Programming for Neural Networks by gnufx on Thursday July 29, 2021 @02:01PM (#27999451)
I took the example program from the OpenMP standard and built it with GCC 11 -Ofast. -fopt-info said the relevant loop was vectorized. Adding -fopenmp gave more vectorization messages from elsewhere, but I don't have time to figure out the difference from the tree dump (not being good with assembler). Doubtless the directives can help, but you do need to get them right, and I trust GCC more than me!
Re:Triton: Open-Source GPU Programming for Neural Networks by einpoklum on Wednesday July 28, 2021 @03:50PM (#27988879)
It's much more limited in expressivity than OpenCL/CUDA, IIANM.
Re:Triton: Open-Source GPU Programming for Neural Networks by 37ef_ced3 on Wednesday July 28, 2021 @03:12PM (#27988302)
Use a domain-specific compiler to generate custom, stand-alone, massively multi-threaded AVX-512 inference C code:

https://NN-512.com

The generated code is easily twice as fast as TensorFlow's AVX-512 kernels (Intel's oneAPI).

Re:Triton: Open-Source GPU Programming for Neural Networks by etaioinshrdlu on Wednesday July 28, 2021 @06:23PM (#27990530)
Your project is well engineered but no matter how many times you post it here, you won’t get real traction without a different development model, such as building on github. This is especially true in a high churn, high cost field like ML. I also think you are being anonymous unnecessarily.
Re:Triton: Open-Source GPU Programming for Neural Networks by johndough on Thursday July 29, 2021 @03:51AM (#27993979)
Some user feedback: I tried to get the ResNet50 example to work, but I gave up after 2 hours. There really should be an end-to-end example like "./resnet50_example.py monkey.jpg".

A few points where I struggled:

- What are the names in the ResNet50Params struct? I looked at a few popular ResNet50 implementations, but I could not find any correspondence and since the names have been sorted, the order of the members might as well be random. I thought about matching the parameters by shape instead, but the chance of getting that correct is almost zero. Why no natural sort? Or even better, a naming scheme that sorts in the same order as execution order, for example with zero-prefixed numbers like "layer05"

- 6 MB C files are somewhat ridiculous. There should at least be a download link somewhere. Copy & pasting from the website would take forever because of scrolling. Also it is impossible to verify that that code monstrosity is not doing anything evil.

- Go seems like an odd choice when almost every machine learning project these days is developed in Python.

- GCC throws a few hundred warnings:

    ResNet50.c: In function ‘ResNet50ThreeArrangeDats7Callee1’:
    ResNet50.c:57009:15: warning: unused variable ‘rel27’ [-Wunused-variable]
    57009 |     ptrdiff_t rel27 = j62-0;
          |               ^~~~~
and

    ResNet50.c: In function ‘ResNet50NetCreate’:
    ResNet50.c:77852:20: warning: taking address of packed member of ‘struct ResNet50Params’ may result in an unaligned pointer value [-Waddress-of-packed-member]
    77852 |             params1->bn1Means,
          |             ~~~~~~~^~~~~~~~~~
- You would get more feedback if there was a platform to discuss such issues (e.g. GitHub) instead of hijacking random threads on HackerNews.

- EDIT: I just realized I could probably just write a parser for that graph file format. This would be much easier if it was something standard like JSON instead.

- EDIT2: I think some shapes in the graph file definition are incorrect (possibly reversed per block?). For example, tensor "one1" should have "ToChannels=64" instead of "ToChannels=256".

Re:Triton: Open-Source GPU Programming for Neural Networks by 37ef_ced3 on Thursday July 29, 2021 @09:48AM (#27996491)
Here is a guy who worked it out:

https://github.com/jonatron/test_nn512

That's a full ResNet50 conversion from the corresponding Caffe protobuf. He also wrote a small script to convert TensorFlow networks.

GCC warns about perfectly valid, correct C. It even warns about switch case fallthough. All of the generated code is fully correct. Turn the warnings off.

Please don't copy and paste from the website! Download the NN-512 executable and run it.

Email me if you still have difficulties.

Re:Triton: Open-Source GPU Programming for Neural Networks by johndough on Thursday July 29, 2021 @12:01PM (#27998017)
Thanks, that code does indeed work! And it is significantly faster than PyTorch (15 vs 60 milliseconds). Maybe I will use it in the future if AVX512 should become more widespread.
Re:Triton: Open-Source GPU Programming for Neural Networks by 37ef_ced3 on Monday August 02, 2021 @04:43PM (#28042363)
The new NN-512 release (version 35) changes the build instructions to suppress GCC's warnings while building the generated code (-w switch):

  // To build an object file:
  // gcc -c -w -std=c99 -pthread -Ofast -mavx512f Example1.c
Thanks for your feedback.
Re:Triton: Open-Source GPU Programming for Neural Networks by 37ef_ced3 on Thursday July 29, 2021 @01:55PM (#27999367)
Awesome. Send me email any time and I will assist you.