Yes. The last time I did this we checked the X-Token header on every request, if it didn't exist or there were multiple we replied 401. If only one was there we checked a DB table of active tokens, if it wasn't there or had expired we replied 401. If it was there but wasn't associated with a role that had access to the requested resource, we replied 403. If it was not expired and had access we continued with the request.

As soon as you get away from "check authentication on every request" your attack vectors increase. As a bad actor, I no longer need to bypass your authentication, I just need to bypass whatever system you have in place to decide whether or not to authenticate me. That's generally going to be easier.