Could it be that they're simply idle sleeping to save power? My first ESPhome device confused me by dropping off and on and it turned out that was the problem, idle sleep was configured by default - the device would wake up, report its status and go back to sleep.

Hmm I don't know. It could be but it's very intermittent. Sometimes it happens a lot, sometimes it doesn't.

I'll try to find out the sleep settings though, thanks for the tip!