I agree of course though I also think there is/was a lot of "low hanging fruit" in inference. Pytorch is really a model training framework and pytorch/python have not traditionally been well suited to fast inference, e.g. with lacking quantization but also lacking optimized inference code. That's changing, for example this recent post: https://pytorch.org/blog/accelerating-generative-ai-2/

But "AI" is still very research focused (training) and not application focused (inference). Llama.cpp et al are filling in this gap.

I honestly would love to full time focus on making high performance tools for ml / applied math, but its a hard area to break into, or at least I’ve been going about it wrong.