No, you don't need to retrieve your keys each year. Before storage, create N transactions moving all your coins to the next derived address. Sign all your transactions at once. Then put it in the safe.

Store the transactions unencrypted on your computer. Send one each year. An attacker can't do anything with them except send them early (and force you to open your safe "sometime within the next year".)