You’re right, but if you substitute the word “spoiled” for “off”, then jt would make sense since no one says “oh this food doesn’t taste spoiled”.