My dad was a music director and assistant pastor. He worked about 5 days a week, and also had another job on the side. Choir practice, church functions and services, preparing sermons, visiting sick church members, counseling members, and doing charity work add up to a good bit of time.

The pastors I'm familiar with all work a typical work week - 40 hours or more. Plus visiting the hospital, counseling, weddings, funerals etc.