I wouldn't say that. I've just tried saying out loud to a friend, and they also said it was weird. I would definitely say "The big red building," or even leave out "big" entirely - it's already been specified.

It's worth knowing that I'm a pure mathematician, and I frequently come out with utterances that others find odd, purely on logical grounds. It's plausible that there are circumstances in which I would violate these rules/guidelines/suggestions/baseless musings/whatever.

That's not one of them.

Why do we privilege individual judgements that something seems "weird," particularly when it would be perfectly understood?

It's as if linguistics is often modeling syntactic rules which reflect something other than natural communication as it actually occurs.