For your task with the 3090, and I assume inference, I would like you suggest do end-to-end on target task.

Sometimes for example, a normalization layer can cancel out/fix an error from a previous layer, then it makes no sense to spend flops to save the numerics.

This is kind of a wild west in inference serving, you sometimes change exp to tanh, sometimes go full 4bit, sometimes remove full layers because you still perform fine on the end-task.

Numerics and correctness are more important for training kernels and backwards passes