Times are just numbers, just shift your work hours accordingly. The only real problem is that the people seeing you leave at 4pm and grumble are the same sort of people who don't acknowledge you starting work at 7am. As long as you don't have those sorts of people around you're fine.