Definitely. I've consistently told recruiters I'm not interested when I hear there's hot-desking involved, and even (politely) declined mid-interview once when I didn't know up-front it was a thing at that particular company.

If your teams have no need to be together physically, then THEY HAVE NO NEED TO COME INTO THE OFFICE.