Good point and definitely possible. In fact I've done it myself - switched computer rejected the old certificate and suddenly nobody could access the app. Fortunately at the time it was only a few people and a beta so not too big of a problem.