Jax Fish House downtown. I used to think it was just a localish place, but I've mentioned 'good oysters in Denver' to vendors who travel a lot and they all immediately say Jax. Admittedly, I haven't been to Jax in quite a few years since moving to the coast, but people I talk with still say it's great.

https://www.jaxfishhouse.com/lodo/