Re:Grid: AI platform from the makers of PyTorch Lightning
by orbifold on Thursday October 08, 2020 @01:51PM
(#24721746)
Might not be your target audience, but high energy physics has been operating an infrastructure like that for years:
https://www.etp.physik.uni-muenchen.de/research/grid-computi...
You can basically use these frameworks to run your analysis jobs on any of the connected HPC centers and interactively move workloads around. The data never has to touch your hard drive either but gets moved to the compute on demand. This is how thousands of physicists do statistical analysis on petabytes of data.
For us is basically integrating clouds directly into your code so the barrier disappears and the cloud providers become an extension of your laptop.