The tones themselves aren't hard (it's mainly just altering the pitch of the sounds you make in different ways); the pronunciation issues come from actual sounds that English speakers just don't make.