The big issue is improvement of standardized work. If you automate early you get tethered by subpar standards. This is natural when you have workers: you iterate work instructions and what tools to use together as a group. This is basically how Toyota is so good at producing: every worker know what to do and when to do it. They haven't always been like this; they have slowly iterated into what they are today.
Improving on work instructions for a robot is tideaus at best, and really, really costly as a worst case: you either re-program, or you're forced to reinvest if the robot isn't capable of the new movements/steps.
Musk also had to learn some lessons Toyota learned years ago, because he trusted a bit too much in automation.
If you want to automate something, you need to know exactly all the corner cases, every step of what the human is doing to make it work. The robots are precise enough, but it's often not that easy to describe the tasks to them.
Think about a company that produces screws, they don't have a program for every possible screw they make. Same goes for a more complicated item like a metal box with a door on it.