My prior would rather be that people have wildly varying sensitivity to caffeine genetically. Some get panic attacks, some don't feel much.

Only true if they drank a cup of coffee since their earliest memories began.

That's not how memory works. You mutate a memory on every recall. Your old memories are all completely changed from what they actually were.

I really only started drinking coffee at my first real job after grad school. They had free coffee in the kitchen, so I'd occassionally have one. Maybe once or twice a week. I was like that for several years, and would occassionally go weeks without a coffee. During that time, I was very productive and went from being a new grad to leading the entire team of veterans in less than five years.

After leaving that job, I now consume fairly regularly (for the past decade at least). I can still easily skip days without coffee, though I do prefer having it daily. I literally see no difference in my day to day between having coffee and not throughout my two decades of experience with coffee. I can just as easily fall asleep after a coffee and I rarely feel amped up from a coffee (if I do, then I just stop drinking it). I've certainly never felt anhedonia like many others have mentioned in the comments when I've taken breaks from coffee.

I think it's clear that people have different experiences with substances. Whether mine is a common one or not, I cannot say. But I do have a baseline to compare to and I can legitimately say the only thing that has ever caused me anhedonia was burning out from too much work (during burn out I was still consuming coffee and it didn't improve my mental state at all).