If done collectively that has worked before. In 2001, almost the entire editorial board of the Springer journal Machine Learning resigned [1] in order to lend their support to a new open-access journal, Journal of Machine Learning Research, which quickly supplanted the former as the top journal in the field. It helped that this included a lot of senior people in the field (Stuart Russell, Geoffrey Hinton, Leslie Kaelbling, etc.) who would be the ones judging ML hiring and tenure cases at many universities.

[1] Resignation letter: http://www.jmlr.org/statement.html