Oh sure, you could go direct to fb. I think the default builds on Raspbian already support the KMS EGL path, though, so in theory it just gives you extra performance at little cost.