You finished a PhD, you should know how much weight his accusations would be given seeing as he isn’t a professor. I kept my mouth shut when I heard my co-advisor was badmouthing my other co-advisor’s research and the result was the committee blowing up at the prospectus. Sometimes it pays to be proactive.