For inference there's a whole spectrum of approaches that let you can trade off flexibility for performance. TF Lite Micro is at one end, hand-written Verilog is at the other.

Typically, flexibility is more important at the start of a project, while deep optimization is more important later. You wanna be able to iterate fast. That said, the flexible approaches are now good enough that you will typically get better ROI from optimizing your model architecture rather than your inference code.

I think the sweet spot today is code-generation, when targeting general purpose cores. There's also increasing numbers of chips with hardware acceleration, which is accessed using a compiler that takes a model architecture as input.