The PETALS project is a step in the right direction. A network of GPUs allows you to do inference (and even retrieve activations!) for the BLOOM model, a model so big it won’t fit on most regular hardware