2) I'm considering taking up Zig as a better (well simpler, because my poor brain can't handle rust) safe language for NIFs. Do you think that zig might make a better fit as it can easily integrate C headers into itself?
3) are dirty nifs not ok?
We use ours for signal processing, not "deep learning" so it may not be completely applicable.
Use this to at least notify the erlang side that it can't do this operation right now, either like send(2) via EAGAIN / EWOULDBLOCK, or, if possible, by implementing a "push-notification" from the CUDA scheduler to the BEAM IO subsystem to try again automatically (sort-of implementing CUDA stream enqueue operations as a custom type of IO) or base a poll(2)-style function (to be called by erlang code that then tries to re-submit or maybe propagate a ressource exhaustion conditon/timeout upstream) on this enqueue-possible notification.
The most important thing to support would be a way to allow the caller to not overfill the queue and allow an erlang thread to efficiently block until woken by a completion event in a CUDA stream.
I'm not sure, but I think the former could be done by manually accounting for queue fill-level in the erlang->CUDA library. The latter should be quite easy with CUDA stream callbacks inserted into a helper stream which got a "block for CUDA stream event $x" inserted right before the stream callback. The benefit would be that it's (mostly?) transparent to potential other Rust/C++ code interacting with the same CUDA streams for e.g. backend-fetching or handling/scheduling expensive CPU processing with tight integration into the CUDA operations.
Last I checked there might not have been an efficient way to syscall-block on CUDA stream events, so that one would have to manually trade between context switch overhead and latency because you'd have (had) to sched_yield(3) in between waking up to check if the stream is done or not. The only blocking wait call was a busy poll that would result in wasting the cpu core and preventing other threads/applications from using it while the GPU was busy going through the queued-up tasks/operations. I didn't try or at least don't remember trying whether stream callbacks could be (ab-)used for delivering progress information without actively/explicitely waiting on any CUDA stream(-event). It might be possible though.
Not crashing is good, but not sufficient to play nicely with the BEAM scheduler. You need to also ensure you're not using up a lot of time in some long-running operation, unless it's a "dirty NIF".