… except for byte-level processing, variable-length codecs, or mixed-precision numerics. That never works with autovectorization and can’t be solved with general-purpose SIMD wrappers. For me the solution was to implement those manually, and even at a scale of just 2 libraries I’ve eneded up with somewhat different project layouts & dispatch mechanisms: https://github.com/ashvardanian/SimSIMD , https://github.com/ashvardanian/StringZilla

One big family not covered there, is sparse data-strictures and related algorithms. I’ve only started integrating scatter/gather in AVX-512 and SVE, and on synthetic benchmarks both look promising: https://github.com/ashvardanian/less_slow.cpp/releases/tag/v...

Those should probably unlock a much wider set of applications for SIMD, but designing libraries for those may benefit from yet another project structure.

Ah yeah, gather/scatter are indeed a rather problematic thing for autovectorization. That said, with no-alias info (which Rust has a lot of) it's possible: https://rust.godbolt.org/z/zTfo9nxhd.

Unfortunately it doesn't get autovectorized without the unsafes, but theoretically it should be possible-ish for bounds checking to be autovectorized (most problematic aspect being that it might be hard to annoying-to-impossible to ensure that in the case of multiple panic/UB sources the proper one happens first).

I'd imagine in any non-trivial situation you'd want a custom layer of abstractions over whatever the language provides for all languages. For that a portable-simd thing is actually a rather good base, on which you could add custom arch-specific abstractions/ops as desired.

Not sure what's problematic with mixed-precision (I know SVE is rather weird for mixed-width elements, but that's about it?), though I primarily don't care about float stuff generally. Also no clue what's problematic with byte-level stuff.

Indeed there are still a bunch of things that you want proper manual SIMD for (hell the SIMDful project I work on has an entire DSL for doing nice SIMD), but autovectorization still covers a good amount.

> except for byte-level processing, variable-length codecs, or mixed-precision numerics. That never works with autovectorization and can’t be solved with general-purpose SIMD wrappers.

Counterexamples: Chromium's byte-level HTML scanning, several var-len bit packing codecs, and Gemma.cpp's matmul is mixed-precision (fp8->bf16->fp32->fp64). All written with the Highway general-purpose SIMD wrapper. Please revise your post or expand upon the structure/dispatch concern.

Interesting references! I remember that Gemma.cpp used Highway, but I haven't checked the others much.

Here is a puzzle, then. Let's say we are checking a single register of bytes for element-wise equality with another register of the same size. In AVX2, the output is another YMM register of 0xFF or 0x00 values. In AVX-512, for full ZMM-wide comparisons, it's a 64-bit mask in the K register.

I struggle to see a good way to abstract such things, even for two consecutive SIMD generations on x86.

A simple enough abstraction is to have the mask type have inputs of both element size and count; so for a ≤256-bit product of those you do the homogeneous-bit elements, and for 512-bit you do the packed-bit values. Conversion methods on the mask can convert those to either an explicit full vector, or a packed bit integer, as you need. (and clang can optimize out unnecessary conversions between the two)

Sure, that would make sense, but in AVX-512, there is also a comparison variant for 2x256-bit inputs that outputs a 32-bit mask and another one for 2x128 inputs and a 16-bit output mask. I would use those in different ways, occasionally mixing with the old AVX2 variant, depending on what I’m doing. I have tried to create a generalizable SIMD framework several times in the past, mostly in 2015-17, and still don’t have a good abstraction even between AVX2 and AVX-512.

You'd just produce the ≤AVX2-style result; clang will switch to using the mask-returning comparison instrs if the usage ends up being a mask (gcc doesn't do such fancy things, so I guess getting the last bits of speedup from this strategy depends on how much control over compiler selection you choose to have).

And if you really care about doing different things for ≤AVX2 vs AVX-512 masks, you're definitely in the "squeeze out the last 1.4x" camp and not the "simple ~8x speedup" one. And, realistically, you won't care about this on every single comparison, so where you really want to explicitly use mask-returning comparisons you could just switch to using intrinsics directly (or a different abstraction over them) temporarily.

As a side-note, mask-returning comparisons have 2x less throughput than homogeneous-bit-returning ones for xmm/ymm as far as uops.info data goes[1] (and use port 5!), so this strategy is kinda just what you want really.

(another strategy can be to just return the arch-specific result type, still preferring the ≤AVX2 style (with, if desired, a separate set of comparison ops for the AVX-512-mask output), and making mask-consuming ops polymorphic over the two)

[1]: https://uops.info/table.html?search=cmpeqb%20ymm%29&cb_lat=o...

Yes indeed, this is what we do :) There is an opaque Mask type for which operations such as CountTrue, AllFalse etc are provided. If you really want the one or other representation, VecFromMask and BitsFromMask/StoreMaskBits convert as required. The former is a no-op on AVX2.

As to the AVX2 comparison, for ICL I think we can do two of those per cycle, so no more than the throughput of an AVX-512 mask comparison. The question is what we do with it afterwards - for VQSort (one of the few applications where comparisons are really the bottleneck?) we are much happier to have the packed bits, because that can feed into vpcompress or a LUT implementing that.

You started with this... take:

> Rust feels like a Python developer’s idea of a high-performance computing language. It’s a great language for many kinds of applications — just not when you need to squeeze out every bit of performance from advanced hardware.

And went on to say that Rust in particular is problematic for:

> byte-level processing

It's particularly odd for you to say this given that the memchr Rust crate is just as fast as stringzilla for substring search. And is generally faster in cases where the needle is invariant, because stringzilla doesn't have APIs for amortizing searcher construction.

We've had a discussion about this before where I provided receipts[1] and we have not had a meeting of the minds on this point. The thing I'm trying to achieve here is to point out that your claims are contested and there is evidence that you're wrong. And so I'd caution readers to also in turn question your higher level claims about Rust being a "Python developer's idea of a high-performance computing language."

[1]: https://old.reddit.com/r/rust/comments/1ayngf6/memchr_vs_str...

Hey! I just mean that there is a very large category of developers, generally coming from the Python world, expecting that switching to Rust is supposed to solve every performance-oriented problem, providing a State-of-the-Art solution magically :)

Sadly, programming doesn't seem to work that way. There are always tradeoffs. Rust does some things very well, but it wouldn't be my first choice for others.

MemChr is a lovely package, and there are a few other really cool SIMD projects in the Rust ecosystem. Regardless, my development velocity for HPC-related projects is higher in C/C++, and I get much more flexibility to leverage newer hardware features, like AMX and SME.

Don't worry his comment is obvious clickbait (complete with a plug for a random blogpost written by him). People that matter (people that are actually writing simd for their day job) can immediately spot a poseur (especially relative to you and Raph).

To be clear, I don't support this. StringZilla is a real and useful project, and its performance is competitive with memchr. They aren't a poser.

(There are reasons to use StringZilla over memchr beyond performance. StringZilla provides a number of interesting string operations beyond just substring search. The memchr crate is far more specialized.)

Thanks, Andrew! Love your work too - keeps me on my toes ;) PS: Hoping to share a few more Rusty bits later this year.

> Do agree that a standard SIMD type is rather pointless, if not immediately, then in like 5 years. (and, seemingly, both Rust and C++ are like over 10 years behind on SIMD, so they're already out-of-date)

What language would you consider to have cutting-edge SIMD support?

I've dipped my toes into SIMD with Rust, [1] on stable with platform-specific intrinsics (SSE2, AVX2, NEON). I would have liked to use stable `std::simd`. I learned that (particularly on AVX2) getting things into the right lanes efficiently is a pain. I would have liked to just use `simd_swizzle!` for that part, and mix that with intrinsics calls. My approach of writing a small C++ or unstable Rust program that does the swizzling and then copying the intrinsics operations it chose into my program's "source" code worked, but I prefer to not have a manual copy'n'paste step between compilation and assembly.

If there's something much better out there in another language, well, I'd be very interested to see it.

[1] I wrote this: https://github.com/infiniteathlete/pixelfmt/blob/main/docs/s...

Don't think any language has standardized SIMD that's particularly nice; Highway is probably a quite nice library on C++, though I haven't used it enough to get comfortable.

The thing I use for my projects is Singeli[1], a DSL specifically made for SIMD stuff (though it's capable of generally sanely doing abstractions over types/operations/loops). Obligatory disclaimer that I'm one of the two people working on its design. It's far from a nice experience starting from nothing, but it's pretty nice for what I do.

Its goal isn't necessarily to unify architectures, but rather make it as easy as possible to make abstractions that do; as such its built-in includes for x86 don't have arbitrary shuffling, but do provide a sane interface over the cases that are supported in a single instruction (not including constant creation/loading), and those can run on NEON unchanged (assuming they're ran on 128-bit vectors, of course, as NEON doesn't support larger ones); and, with Singeli just generating C/C++ currently, you can just map in __builtin_shufflevector if desired. e.g. here's your AVX2 `pre`:

    include 'skin/c' # defines infix a+b & a*b etc to run __add/__mul/... (yes, those aren't here by default, and you can define custom infix/prefix ops)
    include 'arch/c' # defines __add & __mul to do C ops
    include 'arch/iintrinsic/basic' # not necessary for a shuffle, but provides basic x86 arith ops
    include 'arch/iintrinsic/select' # x86 shuffles; there's similar 'arch/neon_intrin/basic' & 'arch/neon_intrin/select' for NEON

    fn pre(inp: [32]i8, out: *[32]i8) : void = {
      store{out, 0, vec_shuffle{16, inp, merge{ # 16 specifies to repeat per 16-elt lane
        range{8}*2+1,            # lower half: 8 Y components; compile-time index calculations
        range{4}*4, range{4}*4+2 # upper half: (4 * U), (4 * V).
      }}}
    }

As a more fancy thing, I've got this working (via bodging together the definitions in CBQN with some sugar to make this pretty; not including all that boilerplate here), compilable to SSE2/AVX2/NEON producing a 4x unrolled core loop, plus tail handling (via reading past the end and doing a load-blend-store if necessary because that's what CBQN's fine with; could easily define a fancy_loop such that it does a scalar tail though). (also can be compiled to RVV via currently-unpublished mappings; need to unroll there; can choose to do either a stripmined loop or one with a separate tail):

    fn sigmoid{E}(r:*E, x:*E, n:ux) : void = {
      def V = arch_preferred_vector{E}
      @fancy_loop{V,4}(r in tup{'dst',r}, x, M in 'mask' over n) {
        if (any_hom{M, ...(x!=x)}) {
          emit{void, 'abort'}
        }
        r{x / __sqrt{1 + x*x}}
      }
      # were it not for a bug in tuple loop var mutation in Singeli having undesired pervasion, this would be possible:
      # @fancy_loop{V,4}(r, x, M in 'mask' over n) {
      #   if (...) ...
      #   r = x / __sqrt{1 + x*x}
      # }
    }
    export{'sigmoid', sigmoid{f32}}

(e.g. generated C for AVX2: https://godbolt.org/z/KTeGsazKP)

[1]: https://github.com/mlochbaum/Singeli

Thanks for posting this, I'll take a look. It wasn't on my radar, but the idea of doing a DSL specifically for SIMD is something I've been thinking about and also starting to explore myself.

Thank you!

> Highway is probably a quite nice library on C++, though I haven't used it enough to get comfortable.

Yeah, I've seen several references to it, including this article, so it's on my list to check out some time.

> The thing I use for my projects is Singeli[1], a DSL specifically made for SIMD stuff

That is...not what I'd imagined you'd say! But I am intrigued.