I don't think time has anything to do with it. Rather it's the solvent in the dry erase marker. I used to use dry erase markers to remove extension numbers from the little tabs on 66 blocks back in the days before IP phones. Sometimes those numbers had been there for 15+ years.

So, my advice would be: if one brand of dry erase marker doesn't remove your permanent marker mark, try a different brand. I think I used Expo.