So in short, yeah, spot on.
I do think tones are a bigger problem than you let on. You can get by without them. I've met people who are pretty fluent except they just have no command of tones, and they can still understand and be understood for the most part. But the intimidation factor is huge, and even if you can be understood, people want to speak well not just adequately.
I think the language would be easier to learn if we'd punt on reading and writing for the first year or two. Every curriculum I've seen incorporates writing heavily from the start. As you note, it's like learning two languages at once. That's a lot of extra effort to put in! I imagine it would be much easier to learn these effectively-two-different-languages serially rather than in parallel. Attain basic proficiency with speaking and listening, then move on to how to read and write what's already known.
我是很酷。
（Wǒ shì hěn kù.）
I am cool.
你是很酷。
（Nǐ shì hěn kù.）
You are cool.
他是很酷。
（Tā shì hěn kù.）
He is cool.
她是很酷。
（Tā shì hěn kù.）
She is cool.
我们是很酷。
（Wǒmen shì hěn kù.）
We are cool.
他们是很酷。
(Tāmen shì hěn kù.)
They are cool.
(1) There are many thousands of characters, so you're effectively learning two languages at once: spoken Mandarin and written Chinese. Reading Chineasy might help you get started won't help much after that. Alphabetic languages have at most a few dozen characters.
(2) There are no cognates. German and Swedish, for example, have lots of cognates recognizable to English speakers.
(3) There are very few loan words. There are lots of English words in Japanese, and lots of Japanese words in English.
(4) There's a huge cultural gap.
(5) There are different languages all called Chinese. Learning Putonghua won't help you much with pre-revolutionary (1911) texts, as they were written in Classical Chinese.
(6) Unless you go to China, the Chinese people you meet will usually speak better English than you will Mandarin, so conversations will lapse into English.
On the plus side: The tones are the least of your problems, and pronouncing Chinese isn't much harder than many other languages. Chinese grammar is dead simple. It has a fixed word order, which is similar to English, and there's no inflexion. I'm also told that once you know a certain number of characters (still several thousand), you can easily guess and infer unfamiliar words. The Practical Chinese Reader series stops using English altogether once you reach Book 5. (It's been superseded by more modern texts.)