1) That's not the most robust solution, especially for deep learning because auto-diff tools don't cross language barriers well
2) If you have an argument for why 1) isn't a big deal, then the next step would be demonstrating why that argument doesn't apply to julia just as well (if not better) than Swift. In fact, Julia has a great interoperability story with other languages, one good example is DiffEqPy and DiffEqR which are Python and R packages respectively for accessing julia's state of the art DifferentialEquations library.
I don't see Tensorflow-Swift going anywhere.
Julia's value proposition is mostly geared towards people trying to do things that are just too difficult, awkward or slow in other languages. It's not just about speed, dynamism and friendly syntax. It's also about solving the expression problem and providing unprecedented composability.
So while it doesn't have great appeal to end users, it does appeal to people who make the sort of things end users want so the community is currently swelling with very talented people making state of the art research libraries. Honestly, whether these libraries attract hordes of 'scripters' or not, doesn't really matter to julia's usefulness to the people who use it currently.
But I think that as julia's package ecosystem evolves and matures (We've only been at 1.0 for a year!), it's going to get more and more attractive to end users.
I cringe a little every time I see a julia advertisement that centres it's premise around "use julia 'cause it's fast!".
I love how fast julia is, but that really isn't the important part.
There is a large subcommunity that doesn't enjoy being forced to drop down to C for anything performance.
Julia already counts some banks on their list, that tend to use ML languages instead of Python due to performance.