Publishers do provide a necessary service: financial support for meetings. One significant problem with "Open Access Everywhere!" is that many conferences are a huge financial net loss even after the registration fees, so they need the publisher's financial support to actually hold a meeting. In this way, most of the smaller conferences that publishers support are subsidized by the heavy hitters. It would be a shame for those smaller meetings to disappear.
It's easier for top tier conferences to choose how to publish. For example, in computer vision, CVPR and ICCV (two top-tier conferences in the field) are currently co-sponsored by IEEE (the publisher) and the Computer Vision Foundation, which is run by well-respected members of the computer vision community. As part of this arrangement, papers from these conferences appear in IEEE Xplore as well as the CVF's open access website, free for everyone: http://www.cv-foundation.org/openaccess/menu.py Of course, only a very small number of heavy-hitting conferences can afford the kind of leverage that CVPR and ICCV needed to pull this arrangement off. In their unique situation, everyone (publishers and academics) benefit from the arrangement. That's not usually the case.
In an open-access-only "Ditch the Publishers!" world, it's unclear how we can continue to support smaller conferences. Not every field has big industry players that can sponsor the meeting, and not many students or advisors have enough grant money to afford increased registration fees.
Most students already can't attend conferences on their own because registration is so expensive. Making it even more so would prevent the students that could benefit the most form attending.
Taxes are connected to the NSF/NIH, which are connected to academics via grants. Then what? Professors should just donate to the conferences they wish to support out of their grant money?
It's a massive racket, the source material (academic papers) are free, the editorial staff are not paid, e-journals have obviated the need for printing and distributing paper copies ... Elsevier and their ilk are a parasite, feeding on the spoils of academic research that is often publicly funded.
But then, in a "publish or die" mentality that researchers are forced into, publishing in journals with higher impact ratings help them keep their job; unless every single researcher agrees to stop publishing in the paid-for journals and move en-masse to open access this sorry situation will continue unabated.