Searching for "TOTP drift" seems to indicate it's a problem. I'm personally not very knowledgeable, but I've used a few TOTP libraries, and I don't remember any actually recording the detected drift like the RFC recommends. At most they allow for some drift from the current time, but that's usually up to 1-1.5mins, so if the token drifts more than that, it won't work.