It's written in the Constitution as to when voting should occur. Remember, this was written for the late 18th century, so doing anything on Sundays was generally frowned upon. Plus, changing something like this would be far more difficult than just giving everyone the day off.

However, in this country there is a constant struggle between trying to let more people vote versus trying to restrict voting. Hence why a lot of people are up in arms about potential voter fraud. The traditional wisdom is that more voters helps one party, fewer helps another. With this as a factor, you can see how it would be very difficult to change the day of voting to make it more convenient for people.