I flatter myself to think that some ideas from that project have lived on in Tensorflow and PyTorch, etc. But the project itself wrapped up when I decided to focus on DL, and it would take a lot of work to bring it back to life.
I think I found it here: https://github.com/bryancatanzaro/copperhead
But I'm not sure what the state is. Looks dead (last commit 8 years ago). Probably just a proof of concept. But why hasn't this been continued?
Blog post and example: https://developer.nvidia.com/blog/copperhead-data-parallel-p... https://github.com/bryancatanzaro/copperhead/blob/master/sam...
Btw, for compiling on-the-fly from a string, I made something similar for our RETURNN project. Example for LSTM: https://github.com/rwth-i6/returnn/blob/a5eaa4ab1bfd5f157628...
This is made in a way that it compiles automatically into an op for Theano or TensorFlow (PyTorch could easily be added as well) and for both CPU and CUDA/GPU.
https://github.com/nv-legate/legate.numpy
Disclaimer: I work on the project they used to do the distributed execution, but otherwise have no connection with Legate.
Edit: And this library was developed by a team managed by one of the original Copperhead developers, in case you're wondering.
https://github.com/rapidsai/cudf/blob/branch-0.20/python/cud...
https://github.com/gmarkall/numba/blob/master/numba/cuda/com...
>but we also need high level expressibility that doesn't require writing kernels in C
the above are possible because C is actually just a frontend to PTX
https://docs.nvidia.com/cuda/parallel-thread-execution/index...
fundamentally you are not going to ever be able to have a way to write cuda kernels without thinking about cuda architecture anymore so than you'll ever be able to write async code without thinking about concurrency.
I worked for a client that had this wonderful Python dsl that compiled to Verilog and VHDL. It was much easier to use than writing the stuff the old way. Much more composable too, not to mention tooling.
They created that by forking an open source project dating back to Python 2.5 that I could never find again.
Imagine if that stuff would still be alive today. You could have a market for paid pypi.org instances providing you with pip installable IP components you can compose and customize easily.
But in this market, sharing is not really a virtue.
Then you have Cudapy : https://github.com/oulgen/CudaPy
But in my opinion, the most future proof solutions are higher level frameworks like Numpy, Jax and Tensorflow. TensorFlow and Jax can JIT compile Python functions to GPU (tf.function).
This blog post is great, and we need these kind of tools for sure, but we also need high level expressibility that doesn't require writing kernels in C. I know there are other projects that have taken up that cause, but it would be great to see NVIDIA double down on something like Copperhead.