Whether or not the courts defer to agencies in different scenarios is not nearly as big a part of "how judicial review works" as it sounds like you're trying to claim. Literally last year the Supreme Court removed a major precedent to how deferential the courts need to be to federal agencies[1], so there's absolutely nothing inherent to judicial review that would require things to continue working the way you claim they do even if it was current precedent.

[1]: https://en.wikipedia.org/wiki/Loper_Bright_Enterprises_v._Ra...