A restaurant is a private company and if a customer is abusive and they don't want to serve them they can completely do it.

The argument is not absurd, it only has its limitations and is not absolute. You have shown one of the limitations: Discrimination based on identity is one of them.