As others have said, it's not as straightforward because the CID contains a hash of the data's encoding which is not the same as the data itself (files are chunked into pieces to facilitate incremental verifiability and better caching, the encoding supports directories, etc.).

Here is IPFS's master issue for "verifiable gateway responses", which summarizes the technical challenges and progress: https://github.com/ipfs/in-web-browsers/issues/128.