Its pretty much the OpenCL-model, except it compiles into AVX2 code / AVX512 code. Very similar to CUDA / OpenCL style programming. Its not single-source like CUDA, but it largely accomplishes the programming model IMO.
Maybe its time I revisit it. I've always thought the OpenMP spec was being written by highly competent programmers. They seem to "get" what is needed. But the question is if I can get my hands on any OpenMP implementation to actually play with.
Not all of us can afford IBM's compiler suite!
But yeah, good enough to play with. But maybe not good enough to achieve high levels of performance. The SIMD stuff is probably simple enough to implement... maybe I should checkout how well LLVM works with OMP SIMD keywords.
I'm looking at the benchmarks I used to look at, and they're all from 2014 or earlier. So maybe I really should double-check modern implementations. We all know GCC 4.x and LLVM 3.x are an eternity ago, so I probably should revisit their performance.
For example: https://www.phoronix.com/scan.php?page=article&item=llvm_cla...
And back then, it was pretty well known that OpenMP implementations were slower than commercial (such as Intel ICC or IBM's OpenMP implementation).
"#pragma omp declare simd" applies over a function call, which then allows that function to be used inside of a "#pragma omp for simd" loop.
A few keywords here and there really help the autovectorizer achieve closer to CUDA-like environments (like... actually having your SIMD code extend "through" a function call, so you can start splitting up the work a bit better).
EDIT: Here's an example from Intel's ICC: https://software.intel.com/content/www/us/en/develop/documen...
The generated code is easily twice as fast as TensorFlow's AVX-512 kernels (Intel's oneAPI).
A few points where I struggled:
- What are the names in the ResNet50Params struct? I looked at a few popular ResNet50 implementations, but I could not find any correspondence and since the names have been sorted, the order of the members might as well be random. I thought about matching the parameters by shape instead, but the chance of getting that correct is almost zero. Why no natural sort? Or even better, a naming scheme that sorts in the same order as execution order, for example with zero-prefixed numbers like "layer05"
- 6 MB C files are somewhat ridiculous. There should at least be a download link somewhere. Copy & pasting from the website would take forever because of scrolling. Also it is impossible to verify that that code monstrosity is not doing anything evil.
- Go seems like an odd choice when almost every machine learning project these days is developed in Python.
- GCC throws a few hundred warnings:
ResNet50.c: In function ‘ResNet50ThreeArrangeDats7Callee1’:
ResNet50.c:57009:15: warning: unused variable ‘rel27’ [-Wunused-variable]
57009 | ptrdiff_t rel27 = j62-0;
| ^~~~~
and ResNet50.c: In function ‘ResNet50NetCreate’:
ResNet50.c:77852:20: warning: taking address of packed member of ‘struct ResNet50Params’ may result in an unaligned pointer value [-Waddress-of-packed-member]
77852 | params1->bn1Means,
| ~~~~~~~^~~~~~~~~~
- You would get more feedback if there was a platform to discuss such issues (e.g. GitHub) instead of hijacking random threads on HackerNews.- EDIT: I just realized I could probably just write a parser for that graph file format. This would be much easier if it was something standard like JSON instead.
- EDIT2: I think some shapes in the graph file definition are incorrect (possibly reversed per block?). For example, tensor "one1" should have "ToChannels=64" instead of "ToChannels=256".
https://github.com/jonatron/test_nn512
That's a full ResNet50 conversion from the corresponding Caffe protobuf. He also wrote a small script to convert TensorFlow networks.
GCC warns about perfectly valid, correct C. It even warns about switch case fallthough. All of the generated code is fully correct. Turn the warnings off.
Please don't copy and paste from the website! Download the NN-512 executable and run it.
Email me if you still have difficulties.
// To build an object file:
// gcc -c -w -std=c99 -pthread -Ofast -mavx512f Example1.c
Thanks for your feedback.