Instead of trying to dedicate time specifically to refactor try to clean up classes as you come across them when fixing bugs or adding features.

Whenever I open a file, I try and take a quick glance to see if there's anything that could use refactoring before working on the actual issue (if it's either a particularly large file or one that hasn't been touched in years I'll check Sonar[1]).

[1] http://www.sonarsource.com/