That you can take something too far in one direction doesn't give reason or explanation for anything except "don't take it too far in that direction". There are probably good explanations why not to take things infinitely far in either direction but they don't really help explain where the line should be drawn.

I think the actual argument here is encoding of seemingly arbitrary colored pictorial combinations overcomplicates character encoding. If you want to display a colored drawing of an arbitrary family SVG is already a thing but if you want to textually encode an arbitrary family you should use characters in your language not expect the text encoding to pick up more arbitrary drawings.

To me, using Emoji was probably a great way to force developer's hands on supporting certain encoding features more complicated languages use even in cases they only wanted to support latin text. That said, this job is already done. We don't need to continue putting everything into ever more complex pictorial encodings via Unicode for the rest of time.