Getting better and better beans is very hard. You’re stuck with what is available. You’re stuck with what water is available (if you’re a reasonable person). You’re stuck with the local roasters.
So of the variables you can control, the grinder is the most important piece of equipment you can buy.
Again. Grinders can make a difference, but they’re the last step in the chain. Do everything else first.
The fact is, there is a certain level of spend needed. If you’re not making espresso, this is very low these days. You can get very cheap, good gear. If you are making espresso, you’re spending several hundred minimum. Although you can lower your spend quite a bit if you’re willing to hand grind for espresso.
As you said, once you are dialed in, really the only variable I am tweaking per bag of beans is the grind.