I've had this issue too, and I think the root cause is that the physical connection between the AirPods and the case can be unreliable/spotty -- likely a light film buildup on the contacts.

I've had success from "massaging" the offending AirPods in a circle pattern while they're in the case -- which I think rubs the contacts together. Anyway, I haven't had the problem for some time now.