No. You are entitled to believe it as a person. But you would have to convince your society about it, if you don't want to be seen with contempt, or if you want that to also have wider appeal.
Notice, though, how you picked a negative example. How about an example that tries (using the same logic) to justify something you already consider bad?
E.g. The same can be said for "racist remarks" or "rape".
One could argue that there is no damage in rape, especially if the victim was drugged and not aware of it happening. Would we be willing to accept that?
How about taking advantage of people to get ahead, including lying et al when you can get away with it? Why would that be wrong "scientifically"? On the contrary, in a game theoretic way it would be totally advantageous for the individual doing it.
At final analysis, morals are things that we don't have to justify scientifically. We might abandon them, or change them, or justify them with regard to other morals and rhetoric, and we might even use some scientific information in the process, but morals are not a subject for scientific assessment.
For example, I think that exposure to black people and the concept of treating the physically/mentally ill damages children. Should I have to justify that?