But you still apply Huffman coding to the bytes after filtering.
I think this is true even when using 16 bits per sample, but i'm not sure.
From what i remember, in gzip, Huffman coding is applied to the offsets of repeated blocks, which are not single bytes.
And it actually does often work differently for PNG! PNG has a handful of preprocessing options for the pixels. So in filter mode 2, for example, deflate is encoding the difference between each pixel and the pixel above it. More or less.