It's definitely not the hardest aspect. Just learn each word and its pronounciation independently. The spelling is just a mnemonic. If you can remember the gender of each word in your native language, or where the stressed syllable is, you can remember how to pronounce each English word as well.

For me the hardest part is using articles (the/a/none) which don't exist in my native language. I still don't understand the exact rules of their usage and usually just insert whichever article sounds the best.