Yes it most certainly does matter! A crop that's harvested even a few days too late is likely to suffer severe nutrient and quality losses.

eta: Of course this varies per crop, so let me back down just a little, but for most food crops there really is a optimal window outside of which the farmers' rather meagre profits evaporate quite fast.