The Python control loop is probably the safest part of this system.

The reason that comma.ai was able to get a working prototype quickly (unlike much bigger and well-funded companies) is that they use an end-to-end deep learning model. The model directly learns to issue commands to the car based on raw image data. The problem with this approach is that neural network models are a blackbox. They can behave in completely unpredictable ways.