My boss and I have a good enough relationship at work where I'm 1000% confident this wouldn't happen, and if said boss did "perceive" some kind of drop in productivity, their response would be to help me with that rather than turn it into a negative. Indeed, during COVID when I expressed to my boss that I was having a tough time, his immediate response was to ask how he could help and to offer me a couple days off to recharge.
And the reason we have that relationship is specifically because we treat each other as friends that also happen to have a working relationship, rather than letting the work relationship dominate all else.
Not everyone will have that kind of relationship with their boss, but I believe it is something everyone should aspire to rather than immediately rejecting it as even a possibility.
also, I would say work is quite a bit different than a gym. it's a lot easier to find a new gym than a new job, and it's much less disruptive to your life if you have to do so.