Yep, and "quality" isn't always the only control you have, i.e. chroma Sub Sampling is an option for JPEG (and maybe others?) as well, which sometimes has to be played around with in my experience if you really care about high-fidelity JPEGs (i.e. full screen photos).

I really wish something existed (or was even built-in to image format encoders in some way, although I recognise that would arguably be bloat) to "bisect" what the optimum settings are per format without degrading quality based off perceptual pixel differences...

I've written a basic thing myself I use to work out the chroma sub sampling (whether 444 is needed, or if 422 can be used) for JPEG encoding photos, and it can do a fair level of "quality" bisection as well, but it's far from perfect, but still useful.

Some of the examples use subsampling, some don't, depending on if I felt it needed it visually. From memory, I think I disabled subsampling for all the illustration images. Except in WebP where you can't disable it. For WebP I used its "sharp YUV" mode instead.

There's a use case where JPEG is better, scroll down to "Progressive rendering" :

https://jakearchibald.com/2020/avif-has-landed/

https://jakearchibald.com/c/progressive-5f94067c.mp4