At a couple other gyms (these were some 20 years ago), you gave your name when you came in, they pulled up your photo, and checked if it was close enough. Again, no need to carry anything.
That requires staff, which is expensive, but my point is to highlight there are options where you don't need to carry anything. Those sound like the world I would prefer.
While a phone may often be more useful, it's not infinitely more useful than other options. When I used to regularly swim at neighborhood pool, which is in walking distance, I had an RFID card to get in. The card could fit in my swim trunks. When the weather was warm enough I could walk to the pool carrying only card, towel, and body wash. I didn't bother getting a locker. A phone would have been less convenient and therefore less useful.
I used to have a ring of like 2-3 dozen keys. Now the only key I need to carry is for my car.
The difference is in that the phone is infinitely more useful and potentially useful; and the dystopia only arises from the fact that it has been used against me in accordance to the surveillance state.
The phone, and the carrying of it, is not really the issue.