Rust is great, but this is an important comment! We used it to implement Weld's compiler and runtime, but we don't expect data scientists who use languages such as Python, Julia, or R to switch over to it; the idea is that these data scientists continue using APIs in these languages, and under the hood, Weld will perform optimizations and compilation for decreasing execution time (and these "under the hood" components are the ones that we wrote in Rust).

Would Weld be able to do a better job if these scientist were using a Rust library instead ?

A lot of people would like to use Rust for data-analysis / machine learning, but there are not really any good batteries-included frameworks for getting started with this.

Lots of languages are well-suited to compilers. See e.g. Haskell accelerate.

I don't really know why you'd use Rust instead of a GC language from the ML family.