Okay, but let's assume you encounter a goat. It is purple. It is from Mexico. It is tiny. It seems to also be a robot. I'm guessing you've never come across someone describing such a thing to you before, such that you can identify from memory the correct way to describe that goat.
So why, when someone shouts out "Look out for that robot purple tiny Mexican goat!" you would immediately peg them as not a native English speaker, but if they had said "Look out for that tiny purple Mexican robot goat!" you would not?
This is not a 'fixed phrase' thing. There are rules, you're just not consciously aware of them.
It's just memory, we recite timeworn phrases because that's what we hear over and over.
Non-native speakers use phrases that "sound wrong" only because they are synthesizing from principle or from their own language, and haven't heard the usual phrases over and over.
This is where rule-based language analysis runs aground.