It doesn't matter that "Swift is faster than Python". Your TensorFlow graph is gonna be slow as all hell if it's running on the CPU. The bottleneck here really isn't performance on a CPU. That's a no-brainer for all non-ML applications.

But for applications that require a GPU (i.e, most ML applications) cutting over to Swift from Python will likely win you nothing in performance, and wouldn't be worth it at all.