Of course. These systems are non-deterministic and you still would require those with domain expertise to extra verify whatever these LLMs output are 300% correct as they do not reason. (Yes, they do not.)

Otherwise, why aren't LLMs and humanoids already replacing human pilots for flying airplanes end-to-end?

Sometimes using a hammer onto every problem isn't a solution; even if the LLM tells you otherwise.