> Splitting the model up between several GPUs would add a third much worse bottleneck – memory bandwidth between the GPUs.

What if you allowed the system to only have a shared memory between every neighboring pair of GPUs?

Would that make sense for an LLM?