Yep, and "quality" isn't always the only control you have, i.e. chroma Sub Sampling is an option for JPEG (and maybe others?) as well, which sometimes has to be played around with in my experience if you really care about high-fidelity JPEGs (i.e. full screen photos).

I really wish something existed (or was even built-in to image format encoders in some way, although I recognise that would arguably be bloat) to "bisect" what the optimum settings are per format without degrading quality based off perceptual pixel differences...

I've written a basic thing myself I use to work out the chroma sub sampling (whether 444 is needed, or if 422 can be used) for JPEG encoding photos, and it can do a fair level of "quality" bisection as well, but it's far from perfect, but still useful.