The concern comes down to how that evaluation of a website is made. It’s the same way people can’t really accept fact checkers because “who fact checks the fact checkers?” With the news ranking it is who evaluates the evaluators. Without an option for users to switch off the part of the algorithm that includes evaluating for journalistic truth (as I will summarize those non partisan factor’s goal) people will always feel like the possibility of manipulation will always be there.