Plants are vanishingly unlikely to actually feel distress, so you are wrong to characterise animal and plant "distress" as the same. They may have survival mechanisms and reactions to being harvested, but this is different from having a central nervous system. There's also no way to know for sure that any given animal does feel distress, but we can make educated judgments, the same as we do for other humans.