It is not justified because however worse the reasons of the other party are, it will never make your own argument true. Your example exhibits the tu-quoque fallacy.

You are right with the last sentence. Unfortunately few people are interested to find out what's true in our world nowadays — for them it is more important to feel justified with their position regardless if its true or false.