Man, you definitely can take 1 hour off in a day 3-4 times a week. Just take this time and go to the gym or do some interval training outside (sprint/job/sprint/jog etc. cross-fit style).

You can sleep a little less if you really need the time, but I'm sure you can take the time to workout... I do take it.