For me, it wasn't the tones or sounds that made Mandarin tricky - once you get used to letting your mouth/tongue make shapes/be places you've trained it not to be the vowle/consonants aren't that bad, and fi you've ever played an instrument/sung, tones are manageable. It was the stupid pictographs. You're literally just memorizing shapes that didn't correspond to anything was the absolute worst. It got better once you learned to recognize the basic forms that were getting composed into parts of other characters, but that really discouraged me/was the hardest part. I basically gave up on written Chinese and focused on pinyin, because that made sense to my head.

Source: took two years of Mandarin and lived in Taiwan for a few months