As with most topics in software engineering I'd say you will be have to keep learning as you go. They keep coming out with larger models that require fancier parallelism and faster data pipelines. Nvidia comes out with a new thing to accelerate inference every year. Want to use something else than Nvidia? Now you need to learn TPU, Trainium, Meta Accelerator (whatever its name is).