I think it depends on your height and bodyweight, tolerance, and how quickly you metabolize caffeine. You also should be counting total mg of caffeine ingested.

I metabolize caffeine slowly and have practically no tolerance. A medium-sized latte anytime after 5 PM will keep me up all night. A decaf equivalent will make it really hard to fall asleep (remember that decaf still has caffeine).

I think my limit when not ingesting caffeine in the early morning is about 200mg before I have issues with falling asleep.