I get that binary format is nice, but I just don't get why instead of adding binary format to an existing good text format Amazon decided to first extend a poor text format and then add binary to that.

Basically: Ion == JSON + extra features + binary format spec. But Ion ~= YAML + binary format spec. You're going to write a new serializer/deserializer in both cases anyway, but in the second one, at least you get the text part for free in almost any language available.