This is definitely the case. I don't trust myself when I've had a few weeks off the bike until I've done some proper practise again. The biggest obvious drop-off is attention - mind-wandering is verboten when riding, and creeps back if you're not used to maintaining concentration. Specific skills like front-brake control also need maintenance, and can always improve, whatever the experience level, as they're pretty subtle. A fortiori for reading the road and driver behaviour.

> I'm less angry at my neighbors with loud bikes riding all the time

Well that's pleasantly tolerant. The consideration should work both ways, but the nature of some of the motorcyclist population doesn't always lend itself that way unfortunately.