Try this model, which uses excludes some of the "ethics" filtering.
Also, IDK what you mean by a "small" GPU, but I am getting ~3.5 tokens/sec on a laptop 2060, and I think I can squeeze more out of it. And there are methods for splitting up the load in small VRAM pools.
Try this model, which uses excludes some of the "ethics" filtering.
Also, IDK what you mean by a "small" GPU, but I am getting ~3.5 tokens/sec on a laptop 2060, and I think I can squeeze more out of it. And there are methods for splitting up the load in small VRAM pools.