I've been learning Mandarin for a long time. I can understand a decent amount and I can speak pretty well, but I can't read much or write worth a damn, and my vocabulary is small. I also speak French, and learning to read and write took basically no time at all (just quickly learn some pretty regular rules, much simpler than English) and a lot of vocabulary carried over.

So in short, yeah, spot on.

I do think tones are a bigger problem than you let on. You can get by without them. I've met people who are pretty fluent except they just have no command of tones, and they can still understand and be understood for the most part. But the intimidation factor is huge, and even if you can be understood, people want to speak well not just adequately.

I think the language would be easier to learn if we'd punt on reading and writing for the first year or two. Every curriculum I've seen incorporates writing heavily from the start. As you note, it's like learning two languages at once. That's a lot of extra effort to put in! I imagine it would be much easier to learn these effectively-two-different-languages serially rather than in parallel. Attain basic proficiency with speaking and listening, then move on to how to read and write what's already known.