xoranth

> Crappy Pixel Fold 2022 mid-range Android CPU

Can you share what LLMs do you run on such small devices/what user case they address?

(Not a rhetorical question, it's just that I see a lot of work on local inference for edge devices with small models, but I could never get a small model to work for me. So I'm curious about other people's user cases.)

Are there any good how-tos for how to set up a non-trivial container with s6 and s6-rc? Last time I looked at this the documentation was pretty sparse, and more of a reference and design document than a set of how-tos.

I believe they mean that since it bypasses the (Tokio) scheduler, so if you use it in async code you lose the main benefit of async code (namely, the scheduler is able to switch to some other task while waiting for IO to complete.). Basically the same behavior you'd get if you called a blocking syscall directly.

On Linux, you might be able to use userfaultfd to make it async...

Thank you for your reply!

> GPUs are also just generally a lot more limiting than SIMD in many other ways.

What do you mean? (besides things like CUDA being available only on Nvidia/fragmentation issues.)

Sure, but how well do they perform compared to vector loads? Do they get converted to vector load + shuffle uops, and therefore require a specific layout anyway?

Last time I tried using gathers on AVX2, performance was comparable to doing scalar loads.

General questions for gamedevs here. How useful is SIMD given that now we have compute shaders on the GPU? If so, what workloads still require SIMD/why would you choose one over the other?

On x86-64, compilers use SIMD instructions and registers to implement floating point math, they just use the single lane instructions. E.g. (https://godbolt.org/z/94b3r8dMn):

    float my_func(float lhs, float rhs) {
        return 2.0f * lhs - 3.0f * rhs;
    }

Becomes:

    my_func(float, float):
        addss   xmm0, xmm0
        mulss   xmm1, DWORD PTR .LC0[rip]
        subss   xmm0, xmm1
        ret

(addss, mulss and subss are SSE2 instructions.)

Sounds a bit like Google's proposal for a `switchto_switch` syscall [1] that would allow for cooperative multithreading bypassing the scheduler.

(the descendants of that proposal is `sched_ext`, so maybe it is possible to implement doors in eBPF + sched_ext?)

[1]: https://youtu.be/KXuZi9aeGTw?t=900

Thanks!

Is the extension you wrote public?

How would this interact with `io_uring`, especially the polling modes (IO_SETUP_SQPOLL, IO_SETUP_IOPOLL)?

I believe the reason they offer no details about how they tuned the kernels is that the tuning is done by a tool provided by AMD. See here:

https://rocm.docs.amd.com/projects/rocBLAS/en/develop/how-to...

Is there any good article on NT internals (that isn't Russinovich' book), that highlight where/how it is better than Linux and other *BSDs?

When asked people point to IOCP vs epoll, but I'm not sure how relevant it is now that Linux has io_uring.

(They also point to stable ABIs for drivers, but I am more interested in internals)

> expression templates

That's one of the cases where you can't "mechanically" translate C++ to Rust. To obtain the same result, a good choice would be a proc macro.

Which is a pain to implement, but will also give you more flexibility.

Thanks!

> That allows things like individual threads to take locks, which is a pretty big leap.

Does anyone know how those get translated into SIMD instructions. Like, how do you do a CAS loop for each lane where each lane can individually succeed or fail? What happens if the lanes point to the same location?

Do you know any good tutorial for ISPC? Documentation is a bit sparse.

It is the same reason in software sometimes you batch operations:

When you add two numbers, the GPU needs to do a lot more stuff besides the addition.

If you implemented SIMT by having multiple cores, you would need to do the extra stuff once per core, so you wouldn't save power (and you have a fixed power budget). With SIMD, you get $NUM_LANES additions, but you do the extra stuff only once, saving power.

(See this article by OP, which goes into more details: https://yosefk.com/blog/its-done-in-hardware-so-its-cheap.ht... )

I believe the author is referring to how many logical threads/hyperthreads can a core run (for AMD and Intel, two. I believe POWER can do 8, Sparc 4).

The extra physical registers are there for superscalar execution, not for SMT/hyperthreading.

People have engineered solutions to make what is available practical (see all the various quantization schemes that have come out).

It is just that there's a limit to how much you can compress the models.

    Location: Europe, CET timezone
    Remote: Yes
    Willing to relocate: Yes (EU/Switzerland)
    Technologies: C++, Python, Rust, Numpy, Pandas, sklearn, numba, SQL, Rust, Assembly.
    Résumé/CV: Upon request.
    Email: [email protected]
    Website: xoranth.net

Experienced Quantitative Researcher with a strong background in writing soft realtime code for trading signals in HFT, as well as building large scale machine learning pipelines. My expertise spans C++, Python, and machine learning, with a proven track record in improving core KPIs.

I'd like to transition to a software developer role _outside the financial industry_ with a focus on low level programming.

Thanks for the clarification!

I believe something like the mechanism you are describing has been in production at Google for at least a decade. See this talk [1].

The kernel interface that the article uses (called sched_ext) is the result of the attempts to mainstream the Google thing.

[1]: https://www.youtube.com/watch?v=KXuZi9aeGTw

> * the whole idea hinges on the compiler being able to figure out the correct instruction schedule ahead of time. While feasible for Intel's/HP's in house compiler team, the authors of other toolchains largely did not bother, instead opting for more conventional code generation that did not performed all too well.

Was Intel's compiler actually able to get good performance on Itanium? How much less screwed would Itanium have been if other toolchains matched the performance on Intel's compiler?

Also, I vaguely remember reading that Itanium also had a different page table structure (like a hash table?). Did that cause problems too?

Dumb question. With io_uring, how do you handle lines that straddle between chunks? I'm asking since, AFAIU, the submitted requests are not guaranteed to be completed in order.

(The easiest I can think of is submitting reads for "overlapped" chunks, I'm not sure there is an easier way and I'm not sure of how much performance overhead there is to it.)

Ok, makes sense. And thanks for the detailed explaination about word boundaries and the hint about the --pcre flag (I hadn't realized it existed).

> Also, \< and \> are in ripgrep 14

Isn't that inconsistent with the way Perl's regex syntax was designed? In Perl's syntax an escaped non-ASCII character is always a literal [^1], and that is guaranteed not to change.

That's nice for beginners because it saves you from having to memorize all the metacharacters. If you are in doubt you on whether something has a special meaning, you just escape it.

[^1]: https://perldoc.perl.org/perlrebackslash#The-backslash

What GCC is doing at -O3 is encoding even (resp. odd) numbers in a bitmap.

43690 = 0xAAAA = 0b1010101010101010

21845 = 0x5555 = 0b0101010101010101

Then

    sal     rax, cl
    test    eax, 43690
    jne     .L3

    // .L3
    mov     edi, OFFSET FLAT:.LC1 // This is a pointer to the string "odd"
    call    puts
    jmp     .L2

is equivalent to C:

    if ((1 << number) & 0xAAAA) {
        puts("odd"); // x64 is little endian
    }

So execution is constant time (it only needs to check against two constants), there's no loop.

It should be advantageous compared to a jump table because it doesn't have to do two indirect jumps (though you now have branches that might mispredict).

Edit: code formatting.

Clang is able to convert the code to a jump table, though GCC can't.

https://c.godbolt.org/z/Wq65j4rrM

> What would it take to make the compiler optimize this code meaningfully?

To make GCC compile the code meaningfully, it suffices to add a bunch of else statements:

https://c.godbolt.org/z/z4KvxnbfG

In fact, GCC gets very clever at -O3.

(what matters is whether the compiler can turn the sequence of ifs into a switch statement, then both compilers have special logic to generate a lookup table.)