One problem relying on fast libraries causes is it makes doing compiler tricks like automatic differentiation (AD) basically impossible. Also, it restricts the types of APIs that make sense. A simple example of this is to compare Scikit learn to Julia. In Scikit learn, most clustering methods don't allow the user to specify a distance function because doing so would require running python inside a tight C loop, tanking performance. In MLJ, on the other hand, basically anything that requires a distance function will allow you to pass one in rather than assuming euclidean distance. This is possible, because a distance function written in Julia can still be fast, so it can be used without slowing down the whole program for people who only want euclidean distances.